Unexpected Routing Behaviour in AWS with VPC Peering and NAT Gateway

VPC Peering Basics

VPC peering is an AWS feature that allows connecting two VPCs on a network level. When VPC peering connection is configured, traffic can be routed between the VPCs using private IP addresses, as if the resources were in the same network. The peered VPCs can reside in the same or different AWS accounts and regions.

VPC Peering Connection (Image source: https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html)

To enable network traffic through the peering connection to both directions, the route tables on subnets of both VPCs need to be configured to route traffic to the peering. As is described in AWS documentation:

To enable private IPv4 traffic between instances in peered VPCs, you must add a route to the route tables associated with the subnets for both instances. The route destination is the CIDR block (or portion of the CIDR block) of the peer VPC and the target is the ID of the VPC peering connection.

(Source: https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-routing.html - highlighting added)

In our research, we identified that this part of the documentation is not true with all configurations.

Findings

In an example environment illustrated below, VPC peering connection is configured between two VPCs. If only a subnet in VPC A have an entry in the route tables routing traffic to the peering connection, hosts in VPC B are not reachable from VPC A with stateful protocols. This is expected, as the routing is stateless and response traffic from VPC B do not have a route back to the VPC A.

Illustration of expected behaviour with routes on only one end of the peering

However, Reversec identified that for an unknown reason, this is not true if VPC A is configured to route the traffic to the peering connection through an internal NAT gateway, like illustrated in the diagram below. With such configuration, the hosts in VPC B are reachable from subnet A2 with stateful protocols, even if there still are no route table entries in VPC B subnets routing response traffic back to the peering connection.

Illustration of unexpected routing behaviour

The behaviour was tested and verified with peering connections within a single account, as well as cross-account. It could be verified that ping and TCP connections from VMs in VPC A to hosts in VPC B work through the peering connection. The only change between the non-reachable and reachable scenarios was the added internal NAT gateway to VPC A.

In the test setup, the route table of the subnet in VPC B had no other routes than the default entry routing traffic inside the VPC. Therefore, the responses for the traffic from VPC A should not be routed back to the peering connection. For the sake of testing, all security groups and NACLs were configured to allow any traffic.

Moreover, it was also identified that if the route table of VPC B subnet had entries for the CIDR of VPC A (10.1.0.0/16) and the target was not the peering connection (pcx-123 in the diagrams) - VPC B hosts were still reachable from VPC A with TCP and ping through the NAT gateway. The matching entries in the route table B were effectively ignored by the traffic when it was response for a connection from the NAT gateway, and it was routed back to the peering connection instead.

These findings are in conflict to what is described in AWS’ documentation quoted earlier.

Impact

The described behaviour can have a security impact when routing configurations are used as a security control.

Consider a scenario where the VPC B has multiple subnets, and resources in only one of them (B1, for example) should be reachable from VPC A. The administrator of VPC B configures the route table of B1 to route traffic back through the VPC peering, and thinks that other subnets in the VPC B are not routing traffic to VPC A.

However, resources in VPC A can still reach all subnets in VPC B with stateful protocols when the traffic is routed through a NAT gateway. This leads to all resources in VPC B being unnecessarily reachable from VPC A, expanding the attack surface at a network level. For an attacker to exploit such a configuration, it would require that the security groups and NACLs attached to resources in VPC B are also allowing excessive traffic from VPC A.

AWS’ Response

As this issue conflicts with the documentation and can have security implications, Reversec reported it to AWS security. The response from AWS, however, stated that it is intended behaviour. Below is a quote from the AWS’ response:

Thank you for bringing your concern to our attention. We have reviewed your findings and can confirm this is expected behavior in certain circumstances involving NAT gateways.

When a NAT gateway is involved on one side of a VPC peering connection, return traffic can flow automatically to the connection initiator even without explicit route table entries. This occurs due to connection tracking and route caching mechanisms that can persist for up to 5 days after being disabled. As best practice, we recommend using Security Groups and Network ACLs as primary security controls rather than relying solely on routing tables for access control.

Conclusions

As per AWS, this is expected behaviour, so customers of AWS need to take it into account when designing the network architecture and security controls in environments involving VPC peering connections. Security Groups and Network ACLs should be the primary methods for limiting the network traffic, and routing configurations should not be considered a security control.

If the general best practices of restricting the network traffic to resources is followed in the environment, the described behaviour has negligible impact. If not, then it is a good idea to review the security group rules of resources in peered VPCs.

The following is a link to a GitHub repository with a Terraform configuration that can be used deploy a proof-of-concept infrastructure with the affected configurations:

https://github.com/ReversecLabs/vpc-peering-nat-example