How We Keep Upgrading Kestra Before 1.0

Kestra's continuous performance improvements before reaching version 1.0 are highlighted by significant enhancements in scheduling and execution throughput, thanks to contributions from the Xiaomi engineering team. The engineering team focused on optimizing the scheduler, which initially took 20 minutes to start with 100,000 flows and triggers, but now takes only 8 seconds due to the implementation of a local cache for triggers. Additionally, changes to the JDBC queue's polling mechanism allow immediate re-polling when results are returned, enhancing processing capabilities and reducing execution latency under high load. Moreover, the Kestra executor's parallelization strategy evolved from running multiple parallel database queries to processing results from a single query in parallel, improving resource utilization and nearly doubling performance. These advancements, largely driven by Xiaomi's production-scale insights, demonstrate a commitment to enhancing Kestra's scalability and efficiency.