LK losses: Training speculative decoding draft models to directly maximize acceptance rate

LK losses are proposed as an alternative training objective to KL divergence for optimizing speculative decoding in large language models (LLMs), aiming to enhance the acceptance rate of draft tokens without computational overhead. This method improves inference throughput across a wide range of model sizes by directly targeting the acceptance rate, which is an important metric in speculative decoding, rather than relying on KL as a proxy measure. Speculative decoding involves a two-stage process where a smaller draft model generates candidate tokens, and a larger target model verifies them, with acceptance rate being crucial for efficiency. LK losses, including a negative log-acceptance objective and a hybrid objective blending KL and TV distance, address the limitations of KL divergence, especially for low-capacity draft models that cannot perfectly match the target distribution. Experiments demonstrate that LK losses consistently outperform KL baselines across different architectures and model sizes, particularly benefiting lower-capacity models and challenging approximation tasks. The approach is scalable, adaptable to various architectures and target models, and has been open-sourced, offering trained models and datasets to the community.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.