Disaggregated Inference, Part 1: When & Where to Route

In this exploration of disaggregated inference for AI/ML tasks, the focus is on optimizing the allocation of GPU resources by separating prefill and decode processes to enhance efficiency and reduce latency. The text discusses the challenges experienced when a single GPU is tasked with both prefill (reading the prompt) and decode (generating tokens), leading to inefficiencies like token jitter. Disaggregation, which involves using separate GPU pools for prefill and decode, is proposed as a solution, offering benefits such as independent scaling and reduced interference, especially for workloads with strict Time Per Output Token (TPOT) requirements. The document outlines when disaggregation is beneficial, highlighting model size and workload characteristics as critical factors, and provides insights into routing requests efficiently across GPU pools. It underscores the importance of cache-aware routing for improving throughput and reducing computational load, with different strategies like NVIDIA’s Dynamo router and DistServe’s placement algorithm mentioned as effective approaches. The text also sets the stage for further discussion on handling the KV cache transfer between GPUs in subsequent parts of the series.