Rearchitecting the Workflows control plane for the agentic era

Workflows, originally designed for human-triggered multi-step applications, has evolved to accommodate a growing demand for agent-triggered workflows operating at machine speed. This shift necessitated the development of a more scalable and robust control plane, leading to the introduction of a new architecture with enhanced concurrency and creation rate limits. The second version of the control plane features two new components, SousChef and Gatekeeper, which manage workflow instances more efficiently by distributing metadata and concurrency slots. The transition from the original system to the new architecture was seamless, allowing for higher throughput and flexibility without downtime. This upgrade supports up to 50,000 concurrent instances and a creation rate of 300 instances per second, ensuring that Workflows can scale effectively with user needs.