November 2026 Summaries
2 posts from Ngrok
Filter
Month:
Year:
Post Summaries
Back to Blog
No summary generated yet.
Nov 16, 2026
3,703 words in the original blog post.
vLLM is an open-source inference server designed to improve the throughput and cost efficiency of serving open-weight large language models, whose token-by-token decoding process often underutilizes GPUs and creates request queues. It addresses compute inefficiency through continuous batching, which replaces completed requests immediately; chunked prefill, which prevents long prompts from blocking active generations; CUDA graphs, which reduce CPU-to-GPU kernel-launch overhead; and optional speculative decoding, which uses fast draft predictions verified by the larger model. Its central memory innovation, PagedAttention, manages growing KV caches in small, noncontiguous blocks rather than reserving worst-case memory for each request, while prefix caching reuses stored computation for shared prompt prefixes and conversation histories. vLLM can further scale through quantization, tensor and pipeline parallelism for models too large for one GPU, expert parallelism for mixture-of-experts models, and data parallelism for higher traffic, often paired with cache-aware routing. Although competing systems such as SGLang offer similar capabilities and may perform especially well for shared-prefix workloads, vLLM emphasizes broad model and hardware support, and many of its core optimizations are enabled by default.
Nov 05, 2026
6,453 words in the original blog post.