# LLM cold starts on HyperPod: from 27 minutes to seconds with model caching

SageMaker HyperPod now pre-loads model weights onto local NVMe and container images onto nodes, with roughly 60% faster scale-out on 57–145 GB models. I walk through migrating a document-analysis endpoint step by step — from the 27-minute time-to-first-token baseline to rewriting the autoscaling policy that only existed to live with the cold start.

- URL: https://fernando.moretes.com/blog/cold-start-de-llm-no-hyperpod-de-27-minutos-a-segundos-com-model-cachi-amazon-sagem

- Markdown: https://fernando.moretes.com/blog/cold-start-de-llm-no-hyperpod-de-27-minutos-a-segundos-com-model-cachi-amazon-sagem/article.md?lang=en

- Published: 2026-09-14T10:15:14.951Z

- Category: AI & Agents

- Tags: SageMaker HyperPod, LLM inference, cold start, KEDA, Karpenter, NVMe, EKS, autoscaling

- Reading time: 7 min

- Source: [Amazon SageMaker HyperPod now supports model caching for faster inference autoscaling and reduced cold starts](https://aws.amazon.com/about-aws/whats-new/2026/09/sgm-hyperpod-model-caching-inf/)

---

An inference pod carrying a 145 GB model took more than 20 minutes between the scheduler accepting it and the first token coming out — and the autoscaler only asked for that pod once the queue was already full. This article tells the migration of an LLM endpoint on SageMaker HyperPod to the model caching AWS released on September 11, 2026: weights on local NVMe, pre-pulled image, and the part no announcement mentions — rewriting the autoscaling policy that had been tuned to live with the cold start.

## The starting point: 27 minutes to first token

The endpoint that threads this story is a document-analysis pipeline — contracts and statements, bursty traffic at the start of business and at end-of-day close. A model with roughly 145 GB of weights on S3, vLLM in a custom 12 GB container on ECR, `p5.48xlarge` instances, an `InferenceEndpointConfig` with an `autoScalingSpec` triggered by CloudWatch invocations.

Measured from `kubectl describe pod`, the time between `Scheduled` and `Ready` split into three slices: 5 to 7 minutes pulling the image from ECR, more than 20 minutes downloading weights from S3 to the node disk, and about 90 seconds loading weights into HBM. The first two slices are network and have nothing to do with the model — they are the same download repeated on every pod, every scale-out, every node replacement by HyperPod's health check.

The operational consequence was not the time itself; it was what we did to live with it. We kept two permanent spare replicas, `scaleUpStabilizationTime` at zero and a low `targetValue` so the autoscaler would anticipate the burst. Two `p5.48xlarge` sitting idle 24 hours a day is 1,440 instance-hours per month bought only to hide a download. The right question was not "how do we make the pod start faster?" — it was "how much of the headroom exists only because the pod is slow?".

## The journey, in the order it was done

1. **Measure the baseline per slice** — Before changing anything, we recorded per pod the image-pull time, the weight-download time and the GPU load time, from pod events and vLLM logs. Without that split there is no way to tell whether a gain came from the cache or from luck.

2. **Confirm NVMe on the instance type** — `weightsCache` requires local NVMe at the `hostPath` (default `/opt/dlami/nvme`). An EBS-only instance does not warn you: the cache never warms and the pod silently falls back to S3. The `p5.48xlarge` ships 8 × 3.84 TB.

3. **Update the Inference Operator add-on** — If `modelCacheConfig` is not recognized, the operator is old. We updated the EKS add-on on a staging cluster first, because the same add-on bundles KEDA and the ALB controller — the update touches more than the cache.

4. **Enable only `imageCache`** — First production change: `imageCache.enabled: true`. It does not alter the inference pod, it only creates a DaemonSet that pre-pulls the image. Measured gain: the 5 to 7 minutes of pull became seconds, in line with the 97% AWS reports.

5. **Enable `weightsCache` and watch the labels** — `weightsCache.enabled: true`, same `hostPath`. We watched `kubectl get nodes --show-labels | grep cache-ready` until every node in the group was labeled before touching the autoscaler.

6. **Version the model path** — The cache does not detect source changes. We started publishing weights under `s3://.../model/v2026-09-11/` and changing the `spec` on every release — the path change is what forces a new warm-up.

7. **Rewrite the autoscaling** — With scale-out in seconds, we removed the two spare replicas, raised `targetValue` and kept `scaleUpStabilizationTime: 0`. The cold start stopped being a hidden parameter inside the policy.

## Two caches, one node label and a preferred affinity

The design has three pieces worth understanding before trusting it.

**Per-node warm-up:** the operator downloads weights from the source (S3, FSx for Lustre, Hugging Face Hub or JumpStart) into a per-deployment isolated directory under the `hostPath`, on every node that satisfies the deployment's scheduling constraints. When it finishes, it applies a node label prefixed `inference.sagemaker.aws.amazon.com/weights-cache-ready.` followed by the configuration's UID; the image cache does the same with `image-cache-ready.` through a DaemonSet.

**Preferred affinity, not required:** the deployment uses *preferred* node affinity on those labels. A pod goes to a warm node when one exists; if none does, it is scheduled anyway and downloads from S3 and ECR as it always did. That is what makes the migration reversible: the worst case is the old behavior, never a pod stuck in `Pending`.

**Read-only mount:** the pod mounts the cache directory as a read-only `hostPath` volume and vLLM reads from local NVMe — AWS quotes about 7 GB/s — instead of crossing the network. The same 145 GB model that took more than 20 minutes to reach disk is now there before the pod exists.

What it does not do: it does not replicate cache across nodes, does not detect new weights under the same key, and does not survive node replacement. A node replaced by the health check is born cold, and the operator has to warm it again while the warm nodes keep serving.

## Model caching lifecycle on HyperPod Inference

From spec apply to ready pod: who warms, who labels, where the fallback kicks in and why a fresh Karpenter node is still cold.

### 🔧 HyperPod Inference Operator

- Controller reads InferenceEndpointConfig (compute)
- Weights warmer per-node download (compute)
- Image DaemonSet pre-pull per node (ci)

### 🟧 AWS — remote sources

- S3 / FSx for Lustre weights v2026-09-11 (storage)
- ECR vLLM image 12 GB (storage)

### 🖥 GPU node — p5.48xlarge

- Local NVMe /opt/dlami/nvme (8 × 3.84 TB) (storage)
- Node label cache-ready.<uid> (network)
- vLLM pod mounts cache read-only (ai)

### 📈 Autoscaling — two layers

- KEDA autoScalingSpec (pods) (compute)
- Karpenter new node = cold cache (compute)

### Flows

- ops -> ctrl: 1. apply spec
- ctrl -> warmer: 2. weightsCache.enabled
- ctrl -> ds: 2. imageCache.enabled
- warmer -> s3: 3. download once per node
- ds -> ecr: 3. pull once per node
- warmer -> nvme: 4. write weights
- warmer -> lbl: 5. label warm node
- keda -> pod: 6. scale-out
- lbl -> pod: 7. preferred affinity
- nvme -> pod: 8. local read ~7 GB/s
- pod -> s3: fallback on cold node
- karp -> pod: new node → slow path

## What changes in autoscaling when the cold start disappears

Autoscaling on HyperPod has two layers, and the cache only acts on one of them.

**Pod layer (KEDA):** the `autoScalingSpec` reads CloudWatch or Amazon Managed Prometheus every `pollingInterval` (default 30 s; the HPA queries KEDA every 15 s) and decides replicas. With a 27-minute cold start, an honest policy has to fire early and keep headroom; that is why we had a conservative `targetValue` and idle replicas. With scale-out in seconds on a warm node, the same policy starts to oscillate — it brings up pods that are ready before the default 300 s `metricCollectionPeriod` window reflects the burst. We lowered `metricCollectionPeriod` to 60 s, raised `targetValue` and left `scaleDownStabilizationTime` at 300 s so capacity is not handed back on the first dip.

**Node layer (Karpenter):** the cache does not help here. A node Karpenter just provisioned has neither weights nor image; the first pod on it falls back and pays the full 27 minutes. If your peak requires new nodes, model caching shortens the second pod on that node, not the first. The decision we made was to keep a floor of warm GPU nodes — the cache warms on them outside peak hours — and use Karpenter only for the rare overflow.

Node-level scale-to-zero and model caching solve different problems. Choosing between them is choosing what you would rather pay for: an idle instance with a ready cache, or a full download in the busiest minute of the day.

## Before and after, in the numbers AWS published and the ones we measured

- **~60%** — faster scale-out. AWS benchmark with weights cache on 57–145 GB models; the gain grows with model size.
- **97%** — less image-pull time. Over 2 minutes removed per pod with the image cache — the lowest-risk change of the migration.
- **1.440 h** — spare instance-hours per month. Two idle p5.48xlarge that existed only to hide the download and left the bill.

## The cost that stays: disk, version and node neighbors

Enabling the cache takes five lines of YAML. Keeping it is what costs, and three things became operating routine.

**NVMe capacity:** every cached deployment occupies its own directory on the same local disk of the node. Two 145 GB models plus a 57 GB one on a `p5.48xlarge` fit comfortably in 30 TB — but the team that consolidates six endpoints onto a smaller instance group discovers disk pressure only when warm-ups stop happening. The documentation is explicit: split instance groups or reduce cached deployments per node.

**Version-based invalidation:** since the source is not monitored, the S3 key is the version. Publishing new weights by overwriting `latest/` leaves the warm node serving the old model indefinitely, with no error at all. A version suffix in the path and a fresh `kubectl apply` is the only reliable trigger; the operator cleans the old files when the old deployment is deleted.

**Fallback signal:** a pod that comes up without the `hostPath` volume is on the slow path and nobody gets told. We built a simple alert: count of nodes carrying the `cache-ready` label lower than the node count of the group for more than 10 minutes, plus per-pod `Scheduled → Ready` time exported to Prometheus. Without it, the regression to the old behavior goes unnoticed until the next peak — and the next peak is exactly when you do not want to find out.

> **The risks we managed along the way:** Three silent failures concentrate the risk of this migration. **Instance without NVMe at the `hostPath`:** the cache never warms, nothing breaks and the cold start stays the same. **Weights overwritten under the same key:** warm nodes serve the old model until someone changes the `spec`. **Fresh Karpenter node at peak:** the first pod pays the full download; the cache only protects the second. None of the three raises an error — all of them produce latency that looks normal.

## Anti-patterns I saw appear in the first week

- **Enabling both caches and the new autoscaler in the same apply**: when `Ready` time drops and the policy oscillates on the same day, nobody knows which change caused what. One change at a time, against a measured baseline.
- **Treating the cache as a replica**: it is per node, not per cluster. Consolidating deployments onto a small instance group to 'share' the cache only shares the disk — and its pressure.
- **Node-level scale-to-zero with model caching as if they were complementary**: when Karpenter removes the last node, it removes the cache with it. The first pod of the day is back to 27 minutes.

> **Curator's note:** I would enable `imageCache` on every inference deployment on HyperPod at the very next change window — it is a DaemonSet, it does not touch the pod, and it hands back two minutes per scale-out asking nothing in return. `weightsCache` I only enable after confirming NVMe on the instance type and putting a version in the S3 path, because the two failures it introduces are mute. The lesson that stays from 16 years operating platforms: every cold-start optimization changes the autoscaling policy that was written to live with the old cold start — if you do not rewrite the policy, you keep paying the old headroom with the new problem already solved.

## Verdict

Use HyperPod model caching when: the model exceeds 50 GB, traffic has predictable bursts, the fleet has local NVMe and you can keep a floor of warm nodes outside the peak. In that scenario it removes from the autoscaler the headroom that existed only to hide the download — and that is where the money shows up, not in the benchmark. Do not expect a gain when the peak requires fresh Karpenter nodes, when the fleet is EBS-only, or when the model fits in a pull of a few minutes: in those three cases the cost of maintaining path versioning and a fallback alert exceeds what the cache gives back. Enable `imageCache` always; enable `weightsCache` under the conditions above.

**Rating:** Recomendado com condições / Recommended 

## References

- [Amazon SageMaker HyperPod now supports model caching for faster inference autoscaling and reduced cold starts (AWS What'](https://aws.amazon.com/about-aws/whats-new/2026/09/sgm-hyperpod-model-caching-inf/)
- [Reduce inference cold starts on Amazon SageMaker HyperPod with model caching (AWS ML Blog, 10 Sep 2026)](https://aws.amazon.com/blogs/machine-learning/reduce-inference-cold-starts-on-amazon-sagemaker-hyperpod-with-model-caching/)
- [Model weights caching and image caching — SageMaker AI Developer Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-model-caching.html)
- [Autoscaling policies for your HyperPod inference model deployment — SageMaker AI Developer Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-autoscaling.html)
- [KV caching and intelligent routing — SageMaker AI Developer Guide](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-model-deployment-caching-routing.html)
- [Amazon SageMaker HyperPod — Generative AI inference architecture and best practices on AWS (Prescriptive Guidance)](https://docs.aws.amazon.com/prescriptive-guidance/latest/gen-ai-inference-architecture-and-best-practices-on-aws/amazon-sage-maker-hyper-pod.html)
- [Unlock efficient model deployment: Simplified Inference Operator setup on Amazon SageMaker HyperPod (AWS Architecture Bl](https://aws.amazon.com/blogs/architecture/unlock-efficient-model-deployment-simplified-inference-operator-setup-on-amazon-sagemaker-hyperpod)
- [Amazon EC2 P5 instances — specifications (NVMe, HBM, EFA)](https://aws.amazon.com/ec2/instance-types/p5/)