LLM Infrastructure Sizing: From Hardware Requirements to Production Capacity

Many VRAM calculators fail to address the critical question of whether a model can handle production traffic, focusing instead on whether a model can load. For instance, a Llama 3.1 70B model requires about 35GB to load at 4-bit quantization, but serving 50 concurrent users with 8K context windows increases memory needs beyond 80GB due to the KV cache, often overlooked in sizing guides. The text explains the complete memory equation necessary for LLM inference, which includes model weights, KV cache, activations, and framework overhead. It emphasizes that production deployments must consider these factors to avoid failures under load, as the KV cache for concurrent requests can exceed available VRAM. The document also provides practical recommendations for infrastructure sizing based on model class, throughput capacity, and cost analysis for self-hosting vs. API use. Additionally, it covers scaling strategies like tensor parallelism, pipeline parallelism, and data parallelism to optimize throughput and cost-efficiency in production environments.