Introducing new Kestra light architecture for open source data orchestration.

Kestra, an open-source orchestration and scheduling platform, has introduced a new lightweight architecture to cater to medium-sized environments, addressing concerns about the complexity of managing Kafka and ElasticSearch for smaller companies. The new version, currently in beta, replaces these technologies with more manageable MySQL or Postgres databases, which are widely used and familiar to many. While this approach maintains most features of the original architecture, it involves trade-offs such as the inability to detect dead workers and limitations on scaling the scheduler beyond a single instance. Additionally, Kestra offers an H2 in-memory or file-based database option for development environments, allowing developers to run Kestra with minimal setup. The Kestra team invites feedback on this beta release to refine and improve the platform further.