LLM cold starts on HyperPod: from 27 minutes to seconds with model caching

The endpoint that threads this story is a document-analysis pipeline — contracts and statements, bursty traffic at the start of business and at end-of-day close. A model with roughly 145 GB of weights on S3, vLLM in a custom 12 GB container on ECR, p5.48xlarge instances, an InferenceEndpointConfig with an autoScalingSpec triggered by CloudWatch invocations.

Measured from kubectl describe pod, the time between Scheduled and Ready split into three slices: 5 to 7 minutes pulling the image from ECR, more than 20 minutes downloading weights from S3 to the node disk, and about 90 seconds loading weights into HBM. The first two slices are network and have nothing to do with the model — they are the same download repeated on every pod, every scale-out, every node replacement by HyperPod's health check.

The operational consequence was not the time itself; it was what we did to live with it. We kept two permanent spare replicas, scaleUpStabilizationTime at zero and a low targetValue so the autoscaler would anticipate the burst. Two p5.48xlarge sitting idle 24 hours a day is 1,440 instance-hours per month bought only to hide a download. The right question was not "how do we make the pod start faster?" — it was "how much of the headroom exists only because the pod is slow?".

The design has three pieces worth understanding before trusting it.

Per-node warm-up: the operator downloads weights from the source (S3, FSx for Lustre, Hugging Face Hub or JumpStart) into a per-deployment isolated directory under the hostPath, on every node that satisfies the deployment's scheduling constraints. When it finishes, it applies a node label prefixed inference.sagemaker.aws.amazon.com/weights-cache-ready. followed by the configuration's UID; the image cache does the same with image-cache-ready. through a DaemonSet.

Preferred affinity, not required: the deployment uses preferred node affinity on those labels. A pod goes to a warm node when one exists; if none does, it is scheduled anyway and downloads from S3 and ECR as it always did. That is what makes the migration reversible: the worst case is the old behavior, never a pod stuck in Pending.

Read-only mount: the pod mounts the cache directory as a read-only hostPath volume and vLLM reads from local NVMe — AWS quotes about 7 GB/s — instead of crossing the network. The same 145 GB model that took more than 20 minutes to reach disk is now there before the pod exists.

What it does not do: it does not replicate cache across nodes, does not detect new weights under the same key, and does not survive node replacement. A node replaced by the health check is born cold, and the operator has to warm it again while the warm nodes keep serving.

Autoscaling on HyperPod has two layers, and the cache only acts on one of them.

Pod layer (KEDA): the autoScalingSpec reads CloudWatch or Amazon Managed Prometheus every pollingInterval (default 30 s; the HPA queries KEDA every 15 s) and decides replicas. With a 27-minute cold start, an honest policy has to fire early and keep headroom; that is why we had a conservative targetValue and idle replicas. With scale-out in seconds on a warm node, the same policy starts to oscillate — it brings up pods that are ready before the default 300 s metricCollectionPeriod window reflects the burst. We lowered metricCollectionPeriod to 60 s, raised targetValue and left scaleDownStabilizationTime at 300 s so capacity is not handed back on the first dip.

Node layer (Karpenter): the cache does not help here. A node Karpenter just provisioned has neither weights nor image; the first pod on it falls back and pays the full 27 minutes. If your peak requires new nodes, model caching shortens the second pod on that node, not the first. The decision we made was to keep a floor of warm GPU nodes — the cache warms on them outside peak hours — and use Karpenter only for the rare overflow.

Node-level scale-to-zero and model caching solve different problems. Choosing between them is choosing what you would rather pay for: an idle instance with a ready cache, or a full download in the busiest minute of the day.

Enabling the cache takes five lines of YAML. Keeping it is what costs, and three things became operating routine.

NVMe capacity: every cached deployment occupies its own directory on the same local disk of the node. Two 145 GB models plus a 57 GB one on a p5.48xlarge fit comfortably in 30 TB — but the team that consolidates six endpoints onto a smaller instance group discovers disk pressure only when warm-ups stop happening. The documentation is explicit: split instance groups or reduce cached deployments per node.

Version-based invalidation: since the source is not monitored, the S3 key is the version. Publishing new weights by overwriting latest/ leaves the warm node serving the old model indefinitely, with no error at all. A version suffix in the path and a fresh kubectl apply is the only reliable trigger; the operator cleans the old files when the old deployment is deleted.

Fallback signal: a pod that comes up without the hostPath volume is on the slow path and nobody gets told. We built a simple alert: count of nodes carrying the cache-ready label lower than the node count of the group for more than 10 minutes, plus per-pod Scheduled → Ready time exported to Prometheus. Without it, the regression to the old behavior goes unnoticed until the next peak — and the next peak is exactly when you do not want to find out.