Scaling Data Operations at Crédit Agricole with Kestra

Julien Legrand from Crédit Agricole details how the bank's data team leverages Kestra to enhance infrastructure management, improve security, and scale data pipelines across over 100 clusters for diverse operations such as NoSQL, MLOps, Streaming, and Big Data. The IT production entity, CAGIP, has transitioned from using Ansible and Jenkins to Kestra, allowing for better parallel processing, resource optimization, and security enhancements like key rotation. After initial experimentation, the team successfully integrated Kestra into their CI/CD workflows, using it for tasks like storing secrets with Vault and managing Git repositories for production flows. They have also linked Kestra to their alerting service for real-time issue notifications. Future plans include using Kestra to replace Kubernetes cron-jobs for data billing, run daily and weekly jobs for platform maintenance, and centralize backup job reports, with ambitions to tackle more complex tasks like certificate management and event-driven processes.