Disaggregated prefill and decode

Large language model inference consists of a compute-intensive prefill phase that processes input tokens and builds KV caches, followed by a memory-bound decode phase that generates output tokens one at a time; combining both workloads on the same hardware can increase decode latency and cause output stutter. Perplexity addresses this through disaggregated prefill and decode, assigning separate nodes to prefill requests and decoder nodes to generate responses after receiving transferred KV caches, improving throughput and latency consistency at the cost of additional time to first token from network transfers. Its KV Messenger coordinates these transfers over RDMA-capable networks using libfabric and fabric-lib, beginning layer-level KV page transfers as soon as they are produced and transferring needed metadata, logits, or hidden states as well. The system supports tensor-parallel and data-parallel deployments, including KV-cache sharding, replication, topology-aware NIC selection, and an HND memory layout that makes sharded cache transfers more efficient. For speculative decoding, the final input token is handled on the decoder to avoid synchronizing sampling and structured-output logic across prefillers. In tests with DeepSeek-R1, separating prefill reportedly raised throughput from inconsistent rates below 50 tokens per second to more than 90 tokens per second with one prefiller supporting three decoder nodes, while deployments of the 480B-parameter Qwen3-Coder model use eight-GPU shards and parallel transfers across associated network interfaces.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.