Aurora

Aurora is an open-source, reinforcement learning-based framework designed to address the limitations of speculative decoding in production environments. It continuously learns and updates from live inference traces, unlike traditional static speculators that often become stale and ineffective as traffic patterns shift. Aurora's design allows it to adapt in real-time across various domains, offering a 1.25x speedup over well-trained static speculators and reducing infrastructure costs by eliminating the need for large-scale offline activation-collection pipelines. The framework supports diverse user demands and is algorithm-agnostic, making it compatible with future speculator designs. Aurora's serve-to-train flywheel approach, powered by RL, ensures efficient, non-disruptive updates, aligning training with real deployment utility rather than just offline quality. Through experiments, Aurora has demonstrated robust online adaptation and performance improvements, challenging the conventional reliance on extensive offline pretraining for speculative decoding.