February 2022 Summaries

Leroy Merlin, a leading brand within the Adeo Group, transitioned its data pipelines from an on-premise setup to a cloud-based system using Kestra to enhance its data operations and meet modern demands. Historically relying on outdated technologies like Teradata and Automic Workload Automation, Leroy Merlin faced challenges such as manual scaling and complex orchestration. The shift to Google Cloud, with a focus on Google BigQuery for storage and Airflow for orchestration, initially presented issues, particularly with Airflow's limitations in handling complex workflows and high CPU usage. Kestra emerged as a viable alternative, allowing for more efficient data ingestion and orchestration through a simple declarative language and custom plugins, streamlining the deployment process and reducing time to market. By embracing Kestra, Leroy Merlin achieved significant improvements in managing its data lifecycle, enabling automation and faster deployment while also contributing to Kestra's development as an open-source project. The transition saw a rapid increase in usage, with Kestra facilitating diverse data operations and supporting the company's growth in cloud-based data processing.

Kestra is an open-source platform designed for orchestrating and scheduling diverse workflows at scale, offering a comprehensive suite of features such as a user-friendly interface, scalability, and extensive plugin support for data orchestration. Originating as a response to the limitations of Apache Airflow during the development of a cloud-based data platform for Leroy Merlin, Kestra was developed over thirty months, integrating technologies like Kafka, ElasticSearch, and Vue.js to address the needs for stability and performance. With its first public release, Kestra marks its readiness for production use, having been effectively implemented at Leroy Merlin since 2020, handling over 350,000 executions monthly. The platform's open-source nature under the Apache License allows for free installation and use, while a forthcoming Enterprise Edition and a software-as-a-service offering aim to enhance its security and high availability. The Kestra team emphasizes community involvement in developing and maintaining plugins, which are crucial for integrating deeply with various tools and databases, eliminating the need for multiple external tools and simplifying the workflow management process.