Architecture, cloud and applied AI blog

LLM cold starts on HyperPod: from 27 minutes to seconds with model caching

SageMaker HyperPod now pre-loads model weights onto local NVMe and container images onto nodes, with roughly 60% faster scale-out on 57–145 GB models. I walk through migrating a document-analysis endpoint step by step — from the 27-minute time-to-first-token baseline to rewriting the autoscaling policy that only existed to live with the cold start.