How a missing S3 gateway endpoint route quadrupled our NAT bill

The finance lead asked why AWS charged us $2,100 for NAT gateway data processing last month. Our normal was around $400. Nothing in the release calendar explained it: no new services, no traffic bump in our own metrics, no scaling events. The bill just quadrupled. Then someone opened VPC Flow Logs and filtered on the NAT's ENI. Roughly 71% of the bytes had destinations in 52.216.0.0/15 and 3.5.0.0/16, which is S3. We had an S3 gateway endpoint on the VPC. It was supposed to be handling that traffic on the AWS backbone for free.

Problem signals:

• NAT gateway data-processing charges (NatGateway-Bytes) climb week over week with no code or workload change
• VPC Flow Logs show heavy traffic to S3 IP ranges (52.216.0.0/15, 3.5.0.0/16) hitting the NAT's ENI instead of the gateway endpoint
• aws ec2 describe-route-tables on a private subnet returns no route for the S3 managed prefix list (pl-xxx)
• NAT gateway BytesOutToDestination stays high while VPC Flow Logs show traffic to the S3 prefix list still routing through NAT (the S3 gateway endpoint itself emits no CloudWatch metrics to check)
• The S3 gateway endpoint exists in the VPC, but the bill still shows $0.045/GB on service-to-service S3 traffic

$412 to $2,103 on a flat workload, all of it NatGateway-Bytes

The bill line item that should not have existed

The NatGateway-Bytes cost for us-east-1 climbed from $412 in March to $2,103 in April. Nothing else on the bill moved. Egress was flat. EC2 was flat. RDS was flat. The whole delta was NAT data processing at $0.045 per GB.

Cost Explorer's usage-type breakdown confirmed the shape. The entire spike was NAT-Bytes on one VPC, in one region, over a four-week ramp. No cliff, no sudden jump, just a slow gradient upward that nobody watched because 'NAT costs a few hundred bucks' was our mental default.

So we mapped the destinations. AWS publishes the IP ranges for S3, DynamoDB, and every other service in ip-ranges.json. We pulled an hour of VPC Flow Logs, joined destination IPs against those ranges, and found the answer. S3 was 71% of the NAT-processed bytes. That should have been impossible. We had a gateway endpoint for S3, and gateway endpoints are free. Their entire point is to keep S3 traffic off NAT.

Cross-region buckets, then a hardcoded public endpoint, then the truth

What we thought first, and why the first two theories died fast

The first theory was that a new batch job was writing to a bucket in a different region. Cross-region S3 traffic does not hit the same-region gateway endpoint; it goes out through NAT. Reasonable theory, wrong theory. We grepped Terraform for cross-region bucket references and found nothing. We checked CloudTrail for recent bucket creations. Nothing new. All our S3 traffic was to buckets in the same region as the workload.

The second theory was that an application was talking to S3 via the public global endpoint URL instead of the regional one. That is a real failure mode: SDKs pointed at a hardcoded https://s3.amazonaws.com sometimes route to the wrong region and skip the gateway endpoint. Also wrong. Every service in the VPC used the SDK default resolver, which picks the regional endpoint.

The third theory turned out to be right. The route table for one of our private subnets was missing the S3 gateway endpoint's prefix-list association. Any pod scheduled onto a node in that subnet was reaching S3 through the NAT, at $0.045 per GB, for six weeks. Same workload the whole time. Different route table.

The route that has to exist, and the day the migration deleted it

How a gateway endpoint quietly stops covering a subnet

A gateway endpoint is not attached to your subnets the way an interface endpoint is. There is no ENI. There is no private DNS record. The gateway endpoint lives in the VPC, and to make it work for a subnet, you have to add its managed prefix list (pl-63a5400a for S3 in us-east-1) as a route in that subnet's route table, with the endpoint as the target.

The whole 'your S3 traffic bypasses NAT' behavior depends entirely on that route existing. If the route is not there, S3 traffic falls through to the default route (0.0.0.0/0), which points at the NAT gateway. There is no error, no warning, no CloudWatch alarm. The traffic just goes through NAT and gets billed per GB.

We checked all four of our private subnets' route tables for the S3 prefix-list route:

aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-0abc123" \
  --query 'RouteTables[].Routes[?DestinationPrefixListId==`pl-63a5400a`]'

Repeat for each private subnet. Empty result means the S3 gateway endpoint is not covering that subnet.

Three subnets returned the S3 prefix-list route. One returned an empty array. That subnet's route table had been rewritten six weeks earlier during a network migration for another team's project. The migration rebuilt the route table from Terraform, but that Terraform module did not know about the gateway endpoint. The endpoint's association was managed by a separate stack. The rewrite silently dropped the S3 route entry.

Nothing failed. Nothing paged. The workload kept running. It just started paying NAT rates for every S3 GET and PUT the pods on those nodes made.

modify-vpc-endpoint, not create-route

The fix, and the command people reach for that does not apply here

The instinct might be to reach for aws ec2 create-route --route-table-id --vpc-endpoint-id to add the endpoint to the table by hand. That does not fit here. Interface endpoints do not use routes at all (they are ENIs with private DNS), and a gateway endpoint's prefix-list route is not added with create-route either. Gateway endpoints get added to a route table by modifying the endpoint itself and telling it which route tables to associate with.

The correct fix is one call:

aws ec2 modify-vpc-endpoint \
  --vpc-endpoint-id vpce-0abc12345 \
  --add-route-table-ids rtb-0def67890

AWS writes the prefix-list route into the route table atomically. No second command.

Verification took thirty seconds:

aws ec2 describe-route-tables --route-table-ids rtb-0def67890 \
  --query 'RouteTables[].Routes[?GatewayId==`vpce-0abc12345`]'

Should now return the pl-xxx prefix-list route with the endpoint as GatewayId.

The route was there. We then pulled a fresh five-minute slice of VPC Flow Logs and joined against S3's IP ranges again. S3 destinations were dropping out of the NAT sample. The subnet was routing them to the gateway endpoint instead. Over the next hour we watched the NAT gateway's BytesOutToDestination CloudWatch metric drop about 40% and level off.

That was the whole fix. Six weeks of overspend, roughly $1,700 in avoidable NAT charges, closed in about four minutes of actual work.

Endpoint-to-route-table coupling in Terraform, plus a rate-of-change NAT alarm

The two things we changed so this stops happening

We changed two things. First, every gateway endpoint in our Terraform is now paired with an explicit list of route-table IDs it associates with, and that list is generated from the same module that generates the private subnets. When someone adds a subnet, the endpoint's association is derived from the same variable. There is no second stack to remember.

Second, we set up a CloudWatch alarm on the NAT gateway's BytesOutToDestination metric with a per-VPC baseline. Not a fixed dollar threshold, a rate-of-change alarm: if NAT bytes for a VPC exceed three times the trailing seven-day median for two consecutive hours, we get paged. We would have caught this ramp on day four instead of week six.

We considered AWS Cost Anomaly Detection. It does catch this shape of spike, and in our archived bills it did fire, eleven days into the ramp. Our own CloudWatch alarm fires in hours because NAT bytes are a real-time metric and the cost bill is not.

For teams reading this with a similar VPC layout, the audit is roughly a ten-minute job. Enumerate every gateway endpoint, enumerate every route table that should be covered, and check for the prefix-list route in each. If you want the deeper cleanup pattern we use for accumulated cloud-cost drift, we have written more of that up in the InfraForge services overview.

This kind of drift does not fail loudly. It just bills.

If your NAT bill just went sideways and nobody deployed anything

The specific shape of this problem is one people miss because it does not fail loudly. Gateway endpoints have no failure mode that produces an error. They just silently stop covering a subnet, and the bill grows. If you have not audited your route tables against your gateway endpoints in the last six months, there is a decent chance one of your subnets is quietly paying NAT rates for S3 or DynamoDB traffic right now.

We have seen this pattern three times this quarter. Two were the same shape as ours: a route-table rewrite that dropped the prefix-list route. One was a subnet added later that never got the association at all. Each was a five-figure annual overrun that took an afternoon to identify and minutes to fix.

If your NAT gateway bill jumped and nobody deployed anything, book an infrastructure review with our team and we will start with a 30-minute diagnostic call this week. We will pull an hour of your Flow Logs, join destination IPs against AWS service ranges, and tell you which subnet is bleeding before the call ends.

Originally published at https://infraforge.agency/insights/nat-gateway-cost-spike-missing-vpc-endpoint-route/.

If your team is dealing with similar infrastructure debt, we offer infrastructure reviews and recovery engagements — see /review.