Cutting the cloud bill without moving anything.

What we walked into.

Honestly, this setup is most companies, and that's not a knock. A few ECS services behind a load balancer, managed Postgres, S3 holding the documents, some batch workers on EC2, and a non-prod environment that was a straight copy of prod. Nothing exotic. The bill was the problem, and only because nobody owned it. Finance would bring it up once a quarter, somebody would go kill whatever looked obvious, and a month later it was right back up. Two years of that.

When we got pulled in, the conversation was already "do we move to a cheaper provider or do we go serverless." Both of those are big projects, and honestly neither one had anything to do with why the bill was high. So we asked for read-only billing access and two weeks before anybody signed off on a migration. That's it, that was the whole ask.

Where the money was actually going.

• The NAT gateway, by a mile. Everything leaving the private subnets for S3 or a third-party API went through it and got billed per GB. Most of that traffic had no reason to leave AWS in the first place.
• Non-prod up 24/7 at full size. The team works maybe 45 hours a week.
• Six or so instances still sized for a launch-day spike from two years ago. Nobody had a reason to look at them again after launch.
• Prod logging at debug, retained forever, in a service that charges you to ingest and then charges you again to keep it.
• A pile of EBS volumes and snapshots attached to nothing, left over from boxes that got shut down over a year ago.
• All of it on-demand. Not a single reservation or savings plan, on a load that hadn't moved in months.

That list is the order we found it in, and it turned out to be roughly the order of cost too. Notice none of it is architecture. It's defaults. Stuff that was the right call on day one and nobody had a reason to come back to.

What we did about it.

NAT went first because it doesn't touch a line of app code. S3 traffic doesn't need to go through a NAT gateway at all, and it's easy to end up there because that's how a VPC comes out of the box. You add a gateway endpoint, it puts a route in your private route tables, and S3 traffic stays inside AWS. Gateway endpoints cost nothing. No hourly rate, no per-GB, nothing.

FIG.01CLICK TO ENLARGE

# S3 gateway endpoint — avoids NAT gateway data fees on S3 traffic.
# Rollback is deleting this resource; the route goes with it.

module "vpc_endpoints" {
  source  = "terraform-aws-modules/vpc/aws//modules/vpc-endpoints"
  version = "6.7.2"

  vpc_id = aws_vpc.main.id

  endpoints = merge(
    {
      s3 = {
        service         = "s3"
        service_type    = "Gateway"
        route_table_ids = [for rt in aws_route_table.private : rt.id]
        tags            = { Name = "s3-gateway" }
      }
    },
    var.uses_dynamodb ? {
      dynamodb = {
        service         = "dynamodb"
        service_type    = "Gateway"
        route_table_ids = [for rt in aws_route_table.private : rt.id]
        tags            = { Name = "dynamodb-gateway" }
      }
    } : {}
  )

  tags = {
    environment = var.environment
    owner       = "platform"
  }
}

One gotcha. The endpoint has to be on every private route table, not just the first one you think of. We missed the batch workers' table on the first pass, and that's exactly where most of the S3 traffic was coming from. The NAT metrics made that obvious inside an hour.

We put one service through it first, watched latency and errors next to the NAT path for a couple days, then moved the rest. It's a five-minute route change but it's still prod, so we treated it like prod. Rollback is deleting one route. Nobody needed a maintenance window for that.

Then non-prod. Everything got an environment tag and a schedule tag. An EventBridge Scheduler rule shuts the tagged instances down and scales the tagged ECS services to zero at night, then brings it all back before anyone's logged on. Weekends it stays off. Somebody needs an environment up for a demo? Tag that one resource schedule=always and it gets skipped. There's no Lambda in the middle to babysit, Scheduler hits the EC2 and ECS APIs directly.

# Stops/starts non-prod on a schedule instead of billing it 24/7.
# Tag a resource schedule=always (e.g. a demo env) to exempt it.

locals {
  tz = var.team_timezone

  # Off 19:00, on 06:30, Mon-Fri. Adjust to your team's hours.
  stop_cron  = "cron(0 19 ? * MON-FRI *)"
  start_cron = "cron(30 6 ? * MON-FRI *)"
}

module "scheduler" {
  source  = "terraform-aws-modules/eventbridge/aws"
  version = "4.3.2"

  create_bus = false # schedules attach to the account's default bus

  schedules = merge(
    {
      "nonprod-stop-workday" = {
        schedule_expression = local.stop_cron
        timezone            = local.tz
        arn                 = "arn:aws:scheduler:::aws-sdk:ec2:stopInstances"
        role_arn            = aws_iam_role.scheduler.arn
        input               = jsonencode({ InstanceIds = data.aws_instances.workday.ids })
      }
      "nonprod-start-workday" = {
        schedule_expression = local.start_cron
        timezone            = local.tz
        arn                 = "arn:aws:scheduler:::aws-sdk:ec2:startInstances"
        role_arn            = aws_iam_role.scheduler.arn
        input               = jsonencode({ InstanceIds = data.aws_instances.workday.ids })
      }
    },
    { for svc, desired in var.workday_ecs_services : "nonprod-stop-${svc}" => {
      schedule_expression = local.stop_cron
      timezone            = local.tz
      arn                 = "arn:aws:scheduler:::aws-sdk:ecs:updateService"
      role_arn            = aws_iam_role.scheduler.arn
      input               = jsonencode({ Cluster = var.nonprod_cluster, Service = svc, DesiredCount = 0 })
    } },
    { for svc, desired in var.workday_ecs_services : "nonprod-start-${svc}" => {
      schedule_expression = local.start_cron
      timezone            = local.tz
      arn                 = "arn:aws:scheduler:::aws-sdk:ecs:updateService"
      role_arn            = aws_iam_role.scheduler.arn
      # desired is what it runs at during the day
      input = jsonencode({ Cluster = var.nonprod_cluster, Service = svc, DesiredCount = desired })
    } }
  )
}

data "aws_instances" "workday" {
  instance_tags = {
    environment = var.environment
    schedule    = "workday"
  }
}

The tagging was the actual work, not the automation. Two services had never been tagged with anything, so they never showed up in a cost report grouped by tag, and the schedule would've skipped right past them. Small thing, but it would've quietly eaten a chunk of the savings and nobody would've known.

Logs were the boring part, which is exactly why they were a problem. Every log group was on never-expire because that's the default and almost nobody changes the default. App logs went down to 30 days. Audit logs went to 400, which is the number compliance gave us once we asked them. We also took prod from debug down to info, because retention does nothing about ingest and ingest was half the log bill.

# Log retention was "never expire" — storage grew for unread logs.
# 30 days for app logs; 400 days for audit logs (compliance's number).

locals {
  app_log_groups = toset([
    "/ecs/api",
    "/ecs/worker",
    "/ecs/web",
    "/aws/lambda/thumbnailer",
  ])

  audit_log_groups = toset([
    "/aws/cloudtrail/org",
    "/app/audit",
  ])
}

module "app_log_group" {
  source   = "terraform-aws-modules/cloudwatch/aws//modules/log-group"
  version  = "5.7.3"
  for_each = local.app_log_groups

  name              = each.value
  retention_in_days = 30

  tags = {
    environment = var.environment
    owner       = "platform"
    retention   = "app"
  }
}

module "audit_log_group" {
  source   = "terraform-aws-modules/cloudwatch/aws//modules/log-group"
  version  = "5.7.3"
  for_each = local.audit_log_groups

  name              = each.value
  retention_in_days = 400

  tags = {
    environment = var.environment
    owner       = "security"
    retention   = "audit"
  }
}

module "log_level" {
  source  = "terraform-aws-modules/ssm-parameter/aws"
  version = "2.1.2"

  name  = "/${var.environment}/api/LOG_LEVEL"
  type  = "String"
  value = var.environment == "prod" ? "info" : "debug"

  tags = {
    environment = var.environment
    owner       = "platform"
  }
}

Rightsizing was last on purpose. Two weeks of real utilization data, then most of the oversized boxes came down a tier or two, and we left headroom on the two services that actually spike. The orphaned volumes and snapshots got a two-week hold with a note on each one saying what it was and when it was going away. Then we deleted them. Nobody said a word.

Where it landed.

NAT spend on the S3 path went to basically zero. Non-prod compute dropped around 70 hours a week. Log storage quit growing forever. We held the reserved-instance conversation until after all of that, on purpose, because you don't lock in a year of capacity you're about to shrink.

All in, the lines we touched came down somewhere between 30 and 50 percent, and egress plus idle non-prod was most of that. We don't put one blended number on this kind of work. It depends entirely on how much of that list is already done before we show up, and on this one, none of it was.

The team runs this same checklist off a monthly calendar invite now. It takes about an hour. It's already caught two smaller regressions, because bills drift exactly the way this one did, one skipped cleanup at a time.

Go check your own account.

• Is traffic to your own cloud's storage going out through a NAT gateway? Put a gateway endpoint in. It's free and it takes five minutes.
• Is anything in non-prod running at 2am on a Sunday? If it is, does it need to be?
• When did anyone last size an instance off real utilization numbers instead of the launch-day guess?
• What's your log retention actually set to? Does anybody read a log older than 30 days?
• Any volumes or snapshots attached to nothing, older than a few months?
• Is your steady-state load reserved, or is every bit of it on-demand?
• Does every resource carry an environment tag and an owner tag? If not, your cost reports aren't telling you the whole story, and you can't fix what you can't see.