Your EC2 Bill is an Architecture Problem, Not an Accounting One

Compute is usually the biggest line on the AWS bill. It’s also the part nobody designs on purpose. Someone picks an x86 On-Demand instance during the first sprint, sizes it “with some headroom,” and that choice quietly sticks around for 2 years.

You can’t fix that from the billing dashboard. Cost is decided much earlier:

1. instance family you pick.
2. how you buy capacity.
3. how your load balancer behaves when a node disappears.
4. where your current containers keep their data.

The above four things decide compute spend and system resilience. This post walks through each one and ends with a working AWS CDK stack that puts it all together.

Pick the hardware for the bottleneck you actually have.

AWS names instances like m5.2xlarge, where m is the class, 5 is the generation, and 2xlarge is the size. Class is the part that matters most, because it sets the ratio of vCPU to memory.

1. General purpose (M, T): about 1 vCPU to 4GiB. Web servers, internal tools, mixed workloads.
2. Compute optimized (C): about 1:2. CPU-heavy APIs, batch jobs, media transcoding, game servers.
3. Memory optimized (R, X): about 1:8. Caches, big JVM heaps, in-memory databases, real-time analytics.
4. Storage optimized (I, D): local NVMe disks with very high random IOPS. Cassandra, ScyllaDB, OpenSearch, and other systems that replicate data themselves.
5. Accelerated (G, P, Inf, Trn): GPUs and AWS’s own ML chips for training and inference.

If your service is CPU-bound, running it on M instances means paying for memory you never touch. If it’s memory-bound, C instances will force you to over-scale. Profile first, then pick.

NOTE:

1. T series can end up costing more than m series if your service runs hot for hours because AWS keeps it fast and bills you for extra credits.
2. If your code doesn’t depend on x86-only binaries, start with Graviton (m8g, c8g, r8g). Graviton4 is up to 30% faster than Graviton3, and Graviton instances are usually priced around 20% below comparable Intel and AMD ones.

How you buy capacity matters as much as what you buy.

EC2 gives you several ways to pay for the same hardware:

• On-Demand: full price, billed per second on Linux, no commitment. Good for unknown traffic and early benchmarking.
• Reserved Instances: up to 72% off for 1 or 3 years, tied to instance type, region, OS, and tenancy. Convertible RIs let you change the type later for a smaller discount (up to about 66%).
• Savings Plans: you commit to a dollar amount per hour (say $10/hr) for 1 or 3 years.
  • EC2 Instance Savings Plans go up to 72% off but lock you to one instance family and region.
  • Compute Savings Plans give a bit less (up to about 66%) but apply across EC2, Fargate, and Lambda, in any region and family. That flexibility is why I'd reach for them first.
  • Anything above your commitment is billed at On-Demand rates.
• Spot: up to 90% off spare capacity, which AWS can take back with a two-minute notice.
• Dedicated Hosts/Dedicated Instances/Capacity Reservations: for licensing, compliance, or guaranteed capacity in one AZ. Capacity Reservations alone give no discount; you pay On-Demand rates whether you use them or not.

The mix that works.

Don't run everything On-Demand. Don't run user-facing APIs 100% on Spot either. The pattern that holds up in production is:

1. Cover your steady 24/7 baseline with a 1-year Compute Savings Plan.
2. Handle daily peaks and burst traffic with Spot, spread across many instance types.
3. Keep a small On-Demand slice as the fallback for when Spot capacity gets tight.

How Spot actually works now.

A lot of people still think of Spot as an auction where someone outbids you and your instance dies. That isn't how it works anymore. Prices now move slowly based on long-term supply and demand. You can still set a max price (it defaults to the On-Demand price), but most interruptions happen because AWS needs the hardware back for On-Demand or Reserved customers in that AZ, not because the price jumped.

Capacity Pools

A Spot capacity pool is one instance type in one Availability Zone. c8g.large in us-east-1a is a different pool from c8g.large in us-east-1b, and both are different from c7g.large in us-east-1a.

If your Auto Scaling Group or Spot Fleet only allows one instance type, you're depending on just a few pools. When those get squeezed, your whole tier goes down at once. So list several families and sizes that your service can run on.

Allocation Strategies

When you give AWS a list of pools, the allocation strategy decides which ones it pulls from:

• lowestPrice: always the cheapest pool. Everyone else picks it too, so it drains fast and you get interrupted more often. Fine for short batch jobs, risky for web traffic.
• diversified: spreads evenly across all pools. Good availability for long-running fleets, but it will use pricier pools you didn't need.
• capacityOptimized: picks the pools with the most spare capacity. Fewest interruptions, sometimes at a slightly higher price.
• priceCapacityOptimized: first narrows to pools with plenty of capacity, then picks the cheapest of those. AWS recommends this one, and it's the right default for most workloads.

Surviving Node Loss: ALB Draining and ECS Storage

With Spot in the mix, nodes will vanish. Users won’t notice if you handle two things well: draining traffic and storing data.

Drain traffic first.

AWS warns you twice before taking a Spot node back:

• Rebalance recommendation: the node is now at higher risk. If Capacity Rebalancing is on, the ASG starts a replacement early.
• Two-minute notice: the node is about to go.

Set ECS_ENABLE_SPOT_INSTANCE_DRAINING=true. When the two-minute notice arrives, ECS moves the node’s tasks to healthy nodes. The ALB stops sending new requests to that node and lets the open ones finish.

The catch is that the ALB waits 300 seconds by default, but Spot only gives you 120. If you leave the default, the node dies mid-drain, and users get 502s. Set the deregistration delay and stopTimeout to 60–90 seconds.

Also use awsvpc mode. Each task gets its own IP, so the ALB sends traffic straight to the task.

Keep data off the node.

Anything written to the node’s own disk is lost when the node goes away. You have three choices:

• Instance Store: very fast, but wiped when the node stops. Use it only for temp files and caches.
• EBS: survives restarts but stays in one AZ and attaches to one node at a time. Use gp3: it’s about 20% cheaper than gp2.
• EFS: shared across AZs and tasks. This is the right pick for Spot, because a task that moves to another AZ still sees the same files.

Putting it together in CDK.

The stack below creates:

• an ECS cluster on a mixed-instance ASG using Graviton families

• one On-Demand base node plus an 80/20 Spot/On-Demand split above it, using PRICE_CAPACITY_OPTIMIZED,

• Spot draining and Capacity Rebalancing,

• an ALB with a deregistration delay that fits inside the Spot window,

• an encrypted EFS volume mounted into the tasks.

import * as cdk from 'aws-cdk-lib'; import * as ec2 from 'aws-cdk-lib/aws-ec2'; import * as ecs from 'aws-cdk-lib/aws-ecs'; import * as iam from 'aws-cdk-lib/aws-iam'; import * as autoscaling from 'aws-cdk-lib/aws-autoscaling'; import * as elbv2 from 'aws-cdk-lib/aws-elasticloadbalancingv2'; import * as efs from 'aws-cdk-lib/aws-efs'; import { Construct } from 'constructs'; export class CostOptimizedEcsClusterStack extends cdk.Stack {   constructor(scope: Construct, id: string, props?: cdk.StackProps) {     super(scope, id, props);     // Three AZs = more Spot capacity pools to choose from.     // One NAT keeps small setups cheap; use one per AZ in production     // to avoid cross-AZ data charges.     const vpc = new ec2.Vpc(this, 'Vpc', { maxAzs: 3, natGateways: 1 });     // Free gateway endpoint: S3 traffic skips the NAT entirely.     vpc.addGatewayEndpoint('S3Endpoint', {       service: ec2.GatewayVpcEndpointAwsService.S3,     });     const cluster = new ecs.Cluster(this, 'Cluster', { vpc });     // Shared storage that survives any single node going away.     const fileSystem = new efs.FileSystem(this, 'SharedFs', {       vpc,       encrypted: true,       performanceMode: efs.PerformanceMode.GENERAL_PURPOSE,       throughputMode: efs.ThroughputMode.ELASTIC,       lifecyclePolicy: efs.LifecyclePolicy.AFTER_30_DAYS, // cold files -> IA     });     // Instance role for the ECS agent.     const instanceRole = new iam.Role(this, 'EcsInstanceRole', {       assumedBy: new iam.ServicePrincipal('ec2.amazonaws.com'),       managedPolicies: [         iam.ManagedPolicy.fromAwsManagedPolicyName(           'service-role/AmazonEC2ContainerServiceforEC2Role',         ),       ],     });     // ARM64 ECS-optimized AMI with a gp3 root volume.     // CDK fills in the ECS agent config when the capacity provider is added.     const launchTemplate = new ec2.LaunchTemplate(this, 'LaunchTemplate', {       machineImage: ecs.EcsOptimizedImage.amazonLinux2023(ecs.AmiHardwareType.ARM),       instanceType: new ec2.InstanceType('c8g.large'),       userData: ec2.UserData.forLinux(),       role: instanceRole,       blockDevices: [         {           deviceName: '/dev/xvda',           volume: ec2.BlockDeviceVolume.ebs(30, {             volumeType: ec2.EbsDeviceVolumeType.GP3,             encrypted: true,           }),         },       ],     });     const asg = new autoscaling.AutoScalingGroup(this, 'Asg', {       vpc,       vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },       minCapacity: 2,       maxCapacity: 20,       capacityRebalance: true, // replace at-risk Spot nodes early       mixedInstancesPolicy: {         launchTemplate,         instancesDistribution: {           onDemandBaseCapacity: 1,                // covered by the Savings Plan           onDemandPercentageAboveBaseCapacity: 20, // 80% Spot above the base           spotAllocationStrategy:             autoscaling.SpotAllocationStrategy.PRICE_CAPACITY_OPTIMIZED,         },         // More types = more pools = fewer correlated interruptions.         launchTemplateOverrides: [           { instanceType: new ec2.InstanceType('c8g.large') },           { instanceType: new ec2.InstanceType('m8g.large') },           { instanceType: new ec2.InstanceType('c7g.large') },           { instanceType: new ec2.InstanceType('m7g.large') },           { instanceType: new ec2.InstanceType('r8g.large') },         ],       },     });     const capacityProvider = new ecs.AsgCapacityProvider(this, 'CapacityProvider', {       autoScalingGroup: asg,       enableManagedTerminationProtection: false,     });     // spotInstanceDraining sets ECS_ENABLE_SPOT_INSTANCE_DRAINING=true on each node.     cluster.addAsgCapacityProvider(capacityProvider, { spotInstanceDraining: true });     const taskDefinition = new ecs.Ec2TaskDefinition(this, 'ApiTask', {       networkMode: ecs.NetworkMode.AWS_VPC,       volumes: [         {           name: 'shared-data',           efsVolumeConfiguration: {             fileSystemId: fileSystem.fileSystemId,             transitEncryption: 'ENABLED',           },         },       ],     });     const container = taskDefinition.addContainer('Api', {       image: ecs.ContainerImage.fromRegistry('public.ecr.aws/nginx/nginx:stable-alpine'),       cpu: 512,       memoryLimitMiB: 1024,       stopTimeout: cdk.Duration.seconds(90), // must fit inside the 2-minute notice     });     container.addPortMappings({ containerPort: 80 });     container.addMountPoints({       sourceVolume: 'shared-data',       containerPath: '/mnt/shared',       readOnly: false,     });     const service = new ecs.Ec2Service(this, 'ApiService', {       cluster,       taskDefinition,       desiredCount: 4,       capacityProviderStrategies: [         { capacityProvider: capacityProvider.capacityProviderName, weight: 1 },       ],     });     fileSystem.connections.allowDefaultPortFrom(service);     const alb = new elbv2.ApplicationLoadBalancer(this, 'Alb', {       vpc,       internetFacing: true,     });     alb.addListener('Http', { port: 80 }).addTargets('Api', {       port: 80,       targets: [service],       deregistrationDelay: cdk.Duration.seconds(60), // default 300s is too long for Spot       healthCheck: {         path: '/health',         interval: cdk.Duration.seconds(15),         healthyThresholdCount: 2,       },     });   } } 

A couple of things worth calling out. spotInstanceDraining: true is the CDK way to set the agent flag, so you don't have to append it to /etc/ecs/ecs.config yourself. And the nodes run in private subnets, so the ALB is the only thing with a public IP.

Costs that eat your savings if you don't look.

Even with the right instances and purchase mix, a few networking line items can take back a big chunk of what you saved.

Cross-AZ traffic and NAT. Data crossing AZs costs $0.01/GB each way. With one NAT Gateway in us-east-1a and nodes spread across three AZs, about two-thirds of your outbound traffic pays that cross-AZ fee on top of the NAT's $0.045/GB processing charge. For real production traffic, run a NAT per AZ. Always add the free gateway endpoints for S3 and DynamoDB so that traffic never touches a NAT.

Public IPv4 addresses. AWS charges $0.005 per hour (about $3.65 a month) for every public IPv4 address, attached or not. Keep cluster nodes in private subnets and clean up unattached Elastic IPs.

The draining mismatch. This one is worth repeating because it's the most common one I see. The ALB's default deregistration delay is 300 seconds. A Spot interruption gives you 120. Leave the default and your users get 502s every time AWS reclaims a node. Set both the deregistration delay and stopTimeout to 60–90 seconds.

Wrapping Up

None of this is complicated on its own. The savings come from treating these choices as design decisions: right-size the instance family, default to ARM, cover your baseline with a Savings Plan, run peaks on diversified Spot, drain cleanly, and keep state off the host. Do that, and the bill goes down, and the system gets better at handling failure at the same time.