Why LLM Inference Needs a New Kind of Router - Part 1

HTTP routing has been a stable field with traditional strategies like round-robin and least-connections effectively balancing traffic across identical backends, but the rise of Large Language Models (LLMs) presents new challenges due to their reliance on GPU pods with unique, stateful characteristics. These models require routers to consider KV cache states, hardware specialization, conversation continuity, and multi-step execution, which are not addressed by traditional stateless routing methods. LLM inference involves maintaining KV caches that drastically affect latency, requiring specialized pods for different phases of processing and ensuring session affinity for conversation continuity. Modular Cloud’s inference framework addresses these complexities through a routing layer that employs a data layer for tracking cache states, a decision layer for routing logic, and an execution layer for multi-step request coordination, using composable plugins to adapt to various deployment patterns without rewriting strategies. This novel approach allows for efficient and effective handling of LLM inference workloads by turning routing decisions into modular, profile-based solutions, paving the way for more flexible and scalable deployments in AI infrastructure.