Cut your vLLM cold starts from 5 minutes to 90 seconds on Runpod Serverless

Cold starts for autoscaled vLLM endpoints combine infrastructure provisioning with model initialization, and testing on Runpod Serverless found that the latter, especially weight loading, torch.compile, and CUDA graph capture, is often the larger source of delay. In tests using a 32B FP8 Qwen model on two H200 GPUs, four configuration changes reduced median end-to-end cold-start delay from 324 seconds to about 91 seconds without affecting request execution latency: persisting the vLLM compile cache on a network volume, prefetching Safetensors weights, limiting CUDA graph capture to realistic batch sizes, and enabling Hugging Face offline mode when all assets are local. Results remain variable because delayTime also includes GPU-capacity waits and uncached image pulls, so repeated trials and log-based verification are needed to isolate configuration effects. The discussion also notes tradeoffs, including network-volume placement constraints, cache invalidation after model or serving-flag changes, potential throughput limits from lowering maximum sequences, and the operational complexity of maintaining caches. FlashBoot, active workers, and longer idle timeouts can reduce how often cold starts occur, while persistent caches help bound worst-case startup time when snapshots miss or new workers must be created.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.