Engineering Den: implementing a new query manager (demo)

The Engineering Den series discusses recent upgrades to the Redpanda Streaming engine and Agentic Data Plane, focusing on improvements to the query manager. Redpanda launched the Agentic Data Plane to connect AI agents with enterprise data securely and acquired Oxla for its distributed SQL Engine capabilities. The old query manager faced issues with state management, leading to queries being stuck in incorrect states and resource allocation problems. To address these, Redpanda developed a new scheduler based on a deterministic state machine that ensures clear state transitions and logs every change, providing enhanced debugging and stability. The new system has successfully run 25,000 queries without the previous issues, demonstrating improved reliability and scalability, and is set for production rollout.