Live draft model training for speculative decoding

Draft models, like EAGLE-3 and DFlash, are increasingly used to enhance large language model (LLM) inference by improving throughput and reducing latency, but aligning these models with base models and dynamic traffic patterns is challenging. A solution has been developed in the form of a distributed training pipeline that uses live inference to extract hidden states and train draft models in real-time, effectively bypassing the need for offline data storage. This approach has led to a median increase in accept rates by 20%, with some traffic patterns experiencing over 100% improvement, translating to faster speculative decoding and more efficient workloads. The architecture, integrated within the Baseten Inference Stack, operates with minimal overhead by using a highly optimized inference engine, leveraging GPU execution, memory management, and networking efficiency. The system also integrates frameworks like UCXX and Trio for robust networking and concurrency management, ensuring resilience against hardware failures and network disruptions.