Deploying LLMs on Kubernetes: vLLM, Ray Serve & GPU Scheduling Guide (2026)

The guide provides a comprehensive overview of deploying and scaling large language models (LLMs) on Kubernetes, emphasizing the intricacies of GPU scheduling, autoscaling, and monitoring. It discusses the deployment of vLLM and Ray Serve, highlighting features like GPU scheduling with MIG, topology awareness, and autoscaling based on queue depth and KV cache utilization. The guide suggests using Kubernetes for LLMs due to its superior handling of GPU workloads and offers configurations verified against specific versions of vLLM and Ray. It also addresses the use of GPU Feature Discovery for efficient scheduling, various serving engines for different scales, and the importance of topology-aware scheduling for latency-sensitive tasks. Additionally, it covers production patterns such as canary rollouts and graceful shutdowns, emphasizing the need to scale based on inference-specific signals rather than traditional CPU metrics. The guide concludes with recommendations for different deployment scenarios, pitfalls to avoid, and monitoring strategies using Prometheus and Grafana, all aimed at optimizing LLM inference on Kubernetes.