Playbook: Where to Run AI on AWS — Lambda vs Fargate vs ECS/EKS (GPU)

The most common confusion I see in AI teams on AWS is treating the system as a monolithic block and trying to fit everything into the same compute. A modern AI system has at least two layers with completely different characteristics:

Orchestration / agent layer: receives the request, builds the prompt, calls tools (search, database, external APIs), manages conversation state, handles retries, logs. This layer is stateless between requests, executes in seconds to a few minutes, and the bottleneck is I/O — not CPU or GPU. Lambda was built for this. An agent waiting for a tool call response is exactly the async wait pattern Lambda handles well, especially with async invocation and Step Functions for longer flows.

Inference / model layer: loads weights (GBs to tens of GBs), runs forward pass on GPU or vectorized CPU, returns tokens. This layer is stateful in memory (weights stay loaded), has latency from hundreds of ms to minutes depending on the model, and the bottleneck is intensive compute. Lambda has no GPU, has a 10 GB RAM limit, and kills the process after 15 minutes — three strikes for any serious model.

When you mix these two layers in the same service, you pay the price of the most expensive layer on every request, including those that don't need it. Separate them. Lambda orchestrates. Bedrock, Fargate, or EKS+GPU serve the model. This separation is the foundation of any AI architecture that scales without invoice surprises.

When someone shows me a spreadsheet comparing Bedrock with self-hosted and concludes self-hosted is 10x cheaper, my first question is: did you include engineering cost?

Bedrock charges per token. You don't manage clusters, configure GPU autoscaling, deal with CUDA drivers, monitor VRAM utilization, or handle model version upgrades. For a team of 3 engineers building a product, the cost of 2 senior engineer weeks configuring EKS+GPU in production can exceed months of token price difference.

The math you need to do:

Total Bedrock cost = tokens × price/token

Total self-hosted cost = 
  (GPU hours × price/hour)
  + (engineering hours setup × cost/hour)
  + (engineering hours maintenance/month × cost/hour × months)
  + (cost of incident when cluster goes down at 3am)

The real break-even depends on your traffic, chosen model, and your team's MLOps maturity. For most early- to mid-stage products, Bedrock wins on total cost even though it's more expensive per token. The inflection point typically appears when you have high and predictable enough traffic to justify Reserved Instances or Savings Plans on GPU, and a dedicated platform team to absorb the operational overhead.

A conservative estimate: if you're generating fewer than 500 million tokens per month with a model equivalent to Claude Haiku or Llama 3 8B, the operational overhead of self-hosted likely negates the token savings. Above 2–5 billion tokens/month with stable traffic, the conversation changes. Do the math with your real numbers.

Pattern 1 — Agent with Bedrock (the most common and correct for 80% of cases)

Lambda receives the user request, builds context, calls bedrock:InvokeModel or bedrock-runtime:InvokeModelWithResponseStream, processes the response, calls tools if needed (other Lambdas, APIs, DynamoDB), and returns. For flows exceeding 15 minutes or with many steps, Step Functions Express Workflows are the natural complement. Cost: Lambda + Bedrock per token. Ops: zero. Time to market: days.

Pattern 2 — RAG Service with Fargate

A RAG service that receives a query, does embedding (small model like all-MiniLM or bge-small), searches OpenSearch/pgvector, does reranking, and returns context. This service runs well on Fargate with 4-8 vCPU and embedding models on CPU. An 80-400 MB embedding model fits comfortably in memory and doesn't need GPU for acceptable latency at moderate throughput. Cost: Fargate per vCPU/hour. Ops: low (ECS service with ALB and CPU-based autoscaling).

Pattern 3 — Self-hosted LLM on EKS+GPU

You have an open-weights model (Llama 3 70B, Mistral, Qwen) and enough traffic to justify the operation. You run vLLM or TGI in pods with nvidia.com/gpu: 1 on EKS, with Karpenter to provision GPU nodes on demand and scale to zero when traffic drops (caution: GPU scale-to-zero has 3-8 minute latency for a new node). You use Spot Instances for batch and On-Demand for real-time. Autoscaling is by custom metric (tokens/second, queue depth) not CPU. Cost: GPU hour + ops. Ops: high. Requires: CUDA drivers in AMI, node selectors/taints in Kubernetes, VRAM monitoring, graceful shutdown to avoid losing in-flight requests.

Pattern 4 — Async Batch Inference

Offline document processing, bulk embedding generation, dataset evaluation. Use SQS + Lambda to orchestrate, and ECS Tasks (not Fargate, use EC2 with GPU) to process each batch. Or use AWS Batch with GPU instances for long-running jobs. Bedrock also has a Batch Inference API for this case without managing anything.