We rebuilt the Kestra engine in 2.0: From Database-Bound Workers to a Decoupled Control Plane

Kestra 2.0 replaces the platform’s prior engine architecture to address database-dependent workers, queue bottlenecks, and the maintenance burden of separate JDBC and Kafka Streams implementations. The new design decouples queue and repository backends, retaining JDBC support for open-source deployments while offering Redis, AMQP, and Kafka queue options in Enterprise, with a single executor, scheduler, and worker implementation across configurations. A new Controller service intermediates communication between workers and backend systems through secured outbound gRPC connections, allowing workers to operate in separate clouds, regions, on-premises environments, or restricted networks without direct database access or exposed inbound ports. The release also reduces queue traffic by loading task outputs only when required, introduces an Indexer Service for non-JDBC backend configurations, supports workload routing through Worker Groups, and enables logs to be stored separately in JDBC, Elasticsearch, Datadog, or Splunk to reduce pressure on primary databases.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.