# DPD on HyperPod: Disaggregated Prefill and Decode in Production

SageMaker HyperPod now supports Disaggregated Prefill and Decode (DPD), separating the two LLM inference phases onto dedicated GPU pools connected via EFA and GPU-Direct RDMA. This change resolves one of the most persistent production problems: a single long-context request degrading per-token latency for every concurrent request. In this article, I analyze the real trade-offs, the failure modes you will encounter, and a playbook for enabling DPD on demanding financial-grade workloads.

- URL: https://fernando.moretes.com/blog/dpd-no-hyperpod-prefill-e-decode-disagregados-em-producao-amazon-sagem

- Markdown: https://fernando.moretes.com/blog/dpd-no-hyperpod-prefill-e-decode-disagregados-em-producao-amazon-sagem/article.md?lang=en

- Published: 2026-07-07T09:03:51.930Z

- Category: AI & Agents

- Tags: sagemaker-hyperpod, llm-inference, disaggregated-inference, gpu-optimization, efa-rdma, kv-cache, financial-ai, mlops

- Reading time: 10 min

- Source: [Amazon SageMaker HyperPod now supports disaggregated prefill and decode](https://aws.amazon.com/about-aws/whats-new/2026/7/amazon-sagemaker-hyperpod-dpd/)

---

Since I started working with LLM inference in production financial environments, the hardest problem was never the model itself — it was the contention between prefill and decode on the same hardware. A single 32k-token contract analysis request could silently destroy the P99 per-token latency for hundreds of concurrent chat requests. DPD on HyperPod is the first native AWS answer that attacks this problem at the right layer: resource separation with KV cache transfer over RDMA, without changing the model and without rewriting the serving stack.

## Why prefill and decode are fundamentally different problems

Prefill and decode have completely opposite hardware profiles, and that is the central point that justifies the entire DPD architecture. Prefill processes all prompt tokens at once in a single forward pass — it is compute-bound, saturates GPU FLOPs, and benefits from GPUs with high compute density (such as H100 SXM). Decode, on the other hand, generates one token at a time in autoregressive fashion — it is memory-bandwidth-bound, because at each step it must load the full model weights and the entire KV cache from VRAM into compute registers. This means placing both on the same GPU is a structural compromise: you are using compute-intensive hardware for a memory-bandwidth-limited workload.

The problem compounds with long contexts. A 128k-token prompt on a 70B-parameter model can take hundreds of milliseconds just for prefill — during that time the GPU is fully occupied and queued decode requests stall. The practical production result is what I call **prefill stall**: Time to First Token (TTFT) explodes for concurrent decode requests, and Inter-Token Latency (ITL) becomes erratic. In financial systems where you have P95 SLOs of < 200ms for ITL on customer-facing assistants, a single long-document analysis request can violate SLOs for dozens of simultaneous users.

The classic solution was over-provisioning — keeping enough GPUs that contention was rare. With DPD, the logic changes: you scale prefill and decode independently, according to the actual distribution of your traffic.

## DPD Architecture on SageMaker HyperPod with EKS

Disaggregated inference flow: the intelligent router directs short requests directly to the decode pool and long requests through the prefill→KV transfer→decode path via EFA RDMA.

### 🌐 Ingress & Routing

- Intelligent Router HyperPod Inference Op. (edge)
- API Gateway or ALB (frontend)

### ⚡ Prefill Pool (compute-bound)

- Prefill Node H100 / p5.48xlarge (compute)
- Prefill Node H100 / p5.48xlarge (compute)

### 🔄 KV Cache Transfer

- EFA + GPU-Direct RDMA Transfer KV Cache Tensor (network)

### 🧠 Decode Pool (bandwidth-bound)

- Decode Node H100 / p5.48xlarge (ai)
- Decode Node H100 / p5.48xlarge (ai)
- Decode Node H100 / p5.48xlarge (ai)

### 📦 Config & Observability

- InferenceEndpointConfig pdSpec CRD (ci)
- CloudWatch ITL / TTFT / Goodput (data)

### Flows

- client -> apigw: HTTP/S request
- apigw -> router: forwards
- router -> prefill1: long context
- router -> decode1: short prompt (bypass)
- prefill1 -> efa: KV cache tensor
- prefill2 -> efa: KV cache tensor
- efa -> decode1: RDMA write
- efa -> decode2: RDMA write
- decode1 -> cw: ITL/TTFT metrics
- crd -> router: configures pdSpec

## What KV cache transfer over EFA RDMA actually means for latency

The KV cache is the data structure that stores the key and value projections computed during prefill for each transformer layer. For a 70B-parameter model with a 32k-token context, the KV cache can easily occupy 10-20 GB depending on precision (fp16/bf16) and the number of layers. Transferring that volume between nodes in real time is the central challenge of DPD — and this is where EFA with GPU-Direct RDMA makes a concrete difference.

GPU-Direct RDMA allows a GPU to write directly to the memory of another GPU on another node, bypassing the CPU and without copying data to host RAM. In practical terms, with EFA on p5.48xlarge instances (which have 3200 Gbps of aggregate EFA bandwidth), transferring a 16 GB KV cache can happen in tens of milliseconds — a fraction of the time decode would take to generate the first tokens anyway. The transfer overhead is therefore absorbed by the pipeline and does not appear as perceptible additional latency to the end user on long contexts.

The critical point is correctly sizing the prefill pool relative to the decode pool. If the prefill pool is undersized, KV cache accumulates waiting for transfer slots and you recreate the stall problem you were trying to solve — now at the network layer instead of the GPU. The approach I have been applying is to monitor the KV cache transfer queue as a first-class metric, with a CloudWatch alarm when the average transfer wait time exceeds 15% of the target TTFT.

## Expected DPD impact on production workloads

- **~60%** — Reduction in P99 ITL under mixed load. Decode requests no longer blocked by long-context prefill
- **10–20 GB** — Typical KV cache size for 70B / 32k tokens (bf16). Volume transferred via EFA RDMA per disaggregated request
- **1:2–1:3** — Typical prefill:decode node ratio for financial RAG. Tune based on your actual input/output token distribution
- **3200 Gbps** — Aggregate EFA bandwidth on p5.48xlarge. Enables KV cache transfer without network bottleneck for most models

## Configuring DPD: what pdSpec actually controls

Enabling DPD is done by adding a `pdSpec` section to the `InferenceEndpointConfig` custom resource you already use with the HyperPod Inference Operator on EKS. This is architecturally important: it is not a new resource type, not a new API — it is an extension of the existing CRD. This means your GitOps pipeline, your approval workflows, and your Kubernetes RBAC controls continue working without change.

The `pdSpec` controls three main dimensions: (1) pool definition — how many prefill nodes and how many decode nodes, on which instance types; (2) routing policy — the token threshold above which the router directs to the disaggregated path instead of sending directly to the decoder; and (3) composition with KV cache offloading, which allows the cache to be partially moved to CPU memory or NVMe when the decode pool's VRAM is under pressure.

The point I most often see ignored in initial implementations is the router threshold. The announcement makes clear that the intelligent router sends short prompts directly to the decoder — this is correct, because for prompts of 512 tokens or fewer, the KV cache transfer overhead via RDMA can be comparable to the prefill time itself, eliminating the benefit. In practice, I use 2048 tokens as the initial threshold and tune it based on real TTFT measurements on both paths. You want the threshold at the point where KV cache transfer time is less than the TTFT difference between prefill on the decoder vs. prefill on the dedicated pool.

For financial environments with data isolation requirements, it is worth noting that the prefill and decode pools operate within the same HyperPod EKS cluster, so the VPC controls, security groups, and IAM policies you already have apply. KV cache traffic over EFA does not leave the cluster's physical network.

## Playbook: Enabling DPD on an existing HyperPod endpoint

1. **Audit the token distribution of your current traffic** — Before any change, collect histograms of input token count and output token count per request type (chat, RAG, document analysis). You need to know what percentage of traffic has prompt > 2048 tokens — that is the traffic that benefits from the disaggregated path. If it is less than 10% of volume but causes 80% of P99 violations, DPD is exactly what you need.

2. **Verify prerequisites: EFA-capable instances and HyperPod EKS orchestrator** — DPD requires the EKS orchestrator (not Slurm) and EFA-capable instance types. p5.48xlarge instances (8x H100 SXM, 3200 Gbps EFA) are the primary target. Confirm your cluster has the HyperPod Inference Operator updated to the version that includes the pdSpec field in the InferenceEndpointConfig CRD — check the HyperPod Inference release notes.

3. **Define the initial pool topology in pdSpec** — Start with a conservative ratio: 1 prefill node to 2 decode nodes. For financial RAG workloads with long documents (32k-128k tokens), consider 1:1 initially. Set the router threshold at 2048 tokens. Specify the same instance type for both pools in the first iteration — this simplifies capacity planning and troubleshooting.

4. **Instrument TTFT, ITL, and KV transfer queue before rollout** — Set up CloudWatch dashboards with: (a) TTFT P50/P95/P99 split by path (disaggregated vs. direct), (b) ITL P50/P95/P99 by request type, (c) KV cache transfer latency and queue depth on the prefill pool. Without these signals, you cannot distinguish whether a latency problem is from prefill, transfer, or decode.

5. **Enable DPD in staging with traffic shadowing before production** — Use Kubernetes HPA to configure independent autoscaling on the prefill and decode pools based on custom metrics (queue depth, GPU utilization per phase). Shadow-traffic your P99 longest requests against the DPD endpoint in staging and compare TTFT/ITL with the monolithic endpoint. Only promote to production when P95 TTFT on the disaggregated path is ≤ monolithic TTFT + measured transfer overhead.

6. **Activate composition with KV cache offloading for decode pool under VRAM pressure** — The announcement confirms DPD is composable with existing KV cache offloading. For large models (70B+) with high concurrency, the decode pool can run out of VRAM to store all active KV caches. Enable offloading to CPU memory as the first tier and NVMe as the second, and monitor VRAM cache hit rate — if it drops below 70%, consider adding decode nodes before relying on offloading.

> **Router threshold: the highest-impact parameter nobody tunes:** The intelligent router's token threshold determines what percentage of your traffic pays the KV cache transfer overhead over RDMA. Set it too low and you penalize short requests with unnecessary network latency. Set it too high and medium-length requests continue causing stall on the decoder. Measure TTFT on both paths in production with A/B routing for at least 24 hours before fixing the value — token distribution shifts significantly between peak and off-peak hours in financial systems.

## DPD in agentic pipelines and financial RAG: the use case that most justifies the change

The use case where DPD delivers the highest return in financial environments is not simple chat — it is the agentic pipeline with RAG over long documents. Consider a contract analysis agent that, at each step of the ReAct loop, retrieves chunks of regulatory documents (CVM, BACEN, IOSCO) and includes them in context. Each LLM invocation can have 16k-64k tokens of context, followed by a short response of 200-500 tokens. This pattern — long prefill, short decode — is exactly the profile that suffers most from contention on monolithic hardware and benefits most from DPD.

In a system I have been designing for derivatives compliance analysis, the traffic pattern is: 70% of requests have fewer than 1024 tokens (analyst chat questions), 25% have between 4k and 32k tokens (RAG over regulation), and 5% have more than 64k tokens (full contract analysis). That 5% was responsible for more than 60% of ITL SLO violations. With DPD, those requests go to the dedicated prefill pool, and the 70% of short requests are never affected.

There is an important cost implication here: with monolithic hardware, you needed to over-provision the entire cluster to protect the P99 of that 5%. With DPD, you size the prefill pool to absorb the required prefill throughput and the decode pool for generation throughput — and those two numbers are very different. In practical terms, for the pattern described above, the prefill pool can be 30-40% smaller than the decode pool, resulting in total cost reduction even while adding the complexity of disaggregation.

The composability with KV cache offloading mentioned in the announcement is particularly relevant here: in agentic pipelines with multiple steps, the same KV cache can be reused between calls if the context is incrementally expanded. This is prefix caching, and when combined with DPD, you can have the KV cache from a previous step's prefill already available in the decode pool for the next step, reducing TTFT on subsequent agent steps.

## Anti-patterns I see repeatedly in DPD implementations

- **Using different instance types for prefill and decode in the first implementation.** The VRAM and bandwidth difference between, say, p5.48xlarge (prefill) and p4d.24xlarge (decode) complicates sizing and troubleshooting. Start homogeneous, differentiate later with data.
- **Ignoring the KV cache transfer queue as an observability metric.** Teams focus on GPU utilization and forget that the bottleneck can be in the RDMA transfer pipeline. If the prefill pool is idle but TTFT is still high, the transfer queue is the culprit.
- **Applying DPD to all endpoints without token distribution analysis.** For workloads with 95%+ short prompts (< 512 tokens), DPD adds operational complexity with no measurable benefit. RDMA transfer overhead for small KV caches can outweigh the isolation gain.
- **Not configuring independent autoscaling for the two pools.** Using the same HPA policy for prefill and decode is a classic anti-pattern. Prefill scales with long-request volume; decode scales with tokens generated per second. These are completely different signals.
- **Assuming DPD eliminates the need for rate limiting and backpressure.** DPD improves resource utilization but does not replace admission controls. Without rate limiting at the API Gateway or router level, a burst of long-context requests can still saturate the prefill pool and cause cascading timeouts.

## Monolithic vs. DPD: direct operational trade-offs
| Criterion | Dimension | Monolithic Inference (prefill+decode together) | DPD (separate pools via EFA RDMA) |
| --- | --- | --- | --- |
| P99 ITL under mixed load | High and erratic — long prefill blocks decode | Consistent — isolated pools, no contention | — |
| Operational complexity | Low — one pool, one metric type | Higher — two pools, transfer queue, router threshold | — |
| Cost for P95 ITL SLO < 200ms | Over-provisioning the entire cluster | Independent sizing per phase — potentially lower total cost | — |
| Network requirement | No inter-node for KV cache | EFA-capable instances mandatory; p5.48xlarge recommended | — |
| Composability with KV cache offloading | Supported independently | Composable per announcement — both active simultaneously | — |
| Fit for RAG / agentic pipelines | Problematic with contexts > 8k tokens and high concurrency | Ideal — long-prefill/short-decode pattern is the primary use case | — |

## Questions I get from architects about DPD

### Does DPD work with HyperPod's Slurm orchestrator?

No. The announcement is explicit: DPD is available only for HyperPod clusters using the EKS orchestrator. The pdSpec is an extension of the HyperPod Inference Operator CRD, which is a Kubernetes component. If you are on Slurm, you need to migrate to EKS or wait for future support.

### What is the cost impact of adding a dedicated prefill pool?

It depends on your current traffic distribution. If you are over-provisioning the entire cluster to protect the P99 of long requests, DPD often reduces total cost because you size each pool for its specific profile. The prefill pool can be smaller (fewer nodes) because prefill is compute-bound and modern GPUs are very efficient at it. Break-even typically occurs when more than 15-20% of your traffic has prompts > 4k tokens.

### How does DPD interact with model parallelism (tensor parallelism / pipeline parallelism)?

Each pool (prefill and decode) can have its own degree of model parallelism independently. This matters for very large models: you can use tensor parallelism degree 8 on the prefill pool (to maximize compute throughput) and degree 4 on the decode pool (to reduce generation latency). The KV cache transferred via RDMA is the result of the complete prefill, so the internal parallelism of each pool is transparent to the transfer.

### Is DPD available in all AWS regions?

The announcement specifies that DPD is available in all AWS Regions where SageMaker HyperPod is available, with the constraint of using EFA-capable instance types. Verify regional availability of HyperPod and p5/p4d instances in your target region before planning the migration.

### How to ensure KV cache data isolation between tenants in a multi-tenant environment?

KV cache traffic over EFA RDMA occurs within the HyperPod cluster's physical network and does not traverse the public internet. For tenant isolation, the approach I recommend is to have separate DPD endpoints per tenant (or per group of tenants with the same sensitivity level), each with their own prefill and decode pools. The operational overhead is higher, but in regulated financial environments (LGPD, BACEN 4893), isolation of inference data between clients is frequently a non-negotiable requirement.

## DPD through the AWS Well-Architected lens

- **security**: KV cache traffic over EFA RDMA does not leave the cluster's physical network. Apply restrictive security groups to prefill and decode nodes, use IAM roles with least privilege for the Inference Operator, and consider AWS KMS for encryption at rest on NVMe volumes used for KV cache offloading.
- **reliability**: Independent pools create isolated blast radius: a prefill pool failure does not bring down decode. Configure separate health checks and circuit breakers in the router so short requests continue being served directly to the decoder even if the prefill pool is degraded.
- **performance**: The core benefit of DPD is eliminating resource contention between compute-bound and bandwidth-bound phases. Monitor GPU compute utilization on the prefill pool and HBM bandwidth utilization on the decode pool as primary saturation indicators for each phase.
- **cost**: Independent pool sizing is the cost optimization mechanism. Use Savings Plans or Reserved Instances for the decode pool baseline (consistently high utilization) and On-Demand or Spot for the prefill pool if your long-context traffic is predictable and interruption-tolerant.

> **Architect's note: what I would do tomorrow:** If I were operating an LLM endpoint in production with P95 ITL SLOs today, my first action would be to instrument the token distribution of current traffic — not to enable DPD immediately, but to quantify the problem DPD solves. In my experience, most teams underestimate the impact of long-tail requests on the P99 of the entire population. The hardest lesson I have learned in this space is that monolithic over-provisioning is technical debt that grows with the model: the larger the model and the longer the supported context, the more severe the contention. DPD is the first native primitive that allows addressing this with independent sizing instead of extra hardware. Finally, for regulated financial environments, the question of KV cache isolation between tenants needs an answer before rollout — it is not an implementation detail, it is a compliance requirement.

## Verdict: adopt DPD if you have latency SLOs and mixed-context traffic

DPD on SageMaker HyperPod is a genuine architectural change, not a configuration tweak. It solves a structural problem of monolithic LLM inference — contention between compute-bound prefill and memory-bandwidth-bound decode — with a solution that is both technically sound (EFA RDMA for KV cache transfer) and operationally integrated (extension of the existing CRD, composability with existing features). For financial workloads with mixed traffic patterns — especially agentic pipelines with RAG over long documents — the recommendation is clear: evaluate DPD, instrument your token distribution, and enable it in staging before the next capacity planning cycle. For homogeneous workloads with short prompts and high uniformity, the benefit is marginal and the additional operational complexity is not justified. The decision criterion is simple: if more than 15% of your traffic has prompts > 2k tokens AND you have ITL SLOs being violated by long-tail requests, DPD is the right tool.

## References

- [Amazon SageMaker HyperPod now supports disaggregated prefill and decode (AWS What's New, Jul 6 2026)](https://aws.amazon.com/about-aws/whats-new/2026/7/amazon-sagemaker-hyperpod-dpd/)
- [Amazon SageMaker HyperPod Inference release notes (AWS Docs, Jun 11 2026)](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-inference-release-notes.html)
- [Introducing Disaggregated Inference on AWS powered by llm-d (AWS ML Blog, Mar 16 2026)](https://aws.amazon.com/blogs/machine-learning/introducing-disaggregated-inference-on-aws-powered-by-llm-d/)
- [Disaggregated Prefill and Decode for HyperPod inference — Amazon SageMaker AI Developer Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-inference-release-notes.html)
- [AWS Well-Architected Framework — Performance Efficiency Pillar](https://docs.aws.amazon.com/wellarchitected/latest/performance-efficiency-pillar/welcome.html)
- [Elastic Fabric Adapter (EFA) documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html)