One token to corrupt them all: a vLLM debugging tale

AI21 Labs encountered a sporadic issue where their Jamba Reasoning 3B model generated gibberish with high confidence during reinforcement learning training. The problem stemmed from the vLLM framework's scheduling and cache management, leading to state corruption under specific runtime conditions. To resolve this, they developed a comparison script to identify and measure failures, ultimately finding that the bug was linked to how vLLM classified new requests due to memory constraints. By tracking request IDs through the forward context and adjusting the classification logic to ensure new requests were always initialized as prefill, they fixed the issue. The debugging process underscored the importance of testing under constrained resources, ensuring determinism, and instrumenting systems for visibility. The lessons learned from this endeavor are applicable to various model architectures and highlight the need for methodical debugging and verification.