How Leroy Merlin managed their cloud data pipelines with Kestra

Leroy Merlin, a leading brand within the Adeo Group, transitioned its data pipelines from an on-premise setup to a cloud-based system using Kestra to enhance its data operations and meet modern demands. Historically relying on outdated technologies like Teradata and Automic Workload Automation, Leroy Merlin faced challenges such as manual scaling and complex orchestration. The shift to Google Cloud, with a focus on Google BigQuery for storage and Airflow for orchestration, initially presented issues, particularly with Airflow's limitations in handling complex workflows and high CPU usage. Kestra emerged as a viable alternative, allowing for more efficient data ingestion and orchestration through a simple declarative language and custom plugins, streamlining the deployment process and reducing time to market. By embracing Kestra, Leroy Merlin achieved significant improvements in managing its data lifecycle, enabling automation and faster deployment while also contributing to Kestra's development as an open-source project. The transition saw a rapid increase in usage, with Kestra facilitating diverse data operations and supporting the company's growth in cloud-based data processing.