October 2024 Summaries

DevOps has revolutionized software development by standardizing processes and enhancing collaboration, yet its principles have not been fully adopted in operational and data workflows, which remain fragmented and siloed due to the use of isolated tools. The proposed solution is an "Ops-Everything" model that integrates all operational workflows into a unified platform, supporting open standards and scalable orchestration akin to DevOps. This model promises to centralize visibility, enable real-time monitoring, and improve collaboration by offering a comprehensive orchestration layer that synchronizes data and operational processes, from ingestion to deployment. Kestra exemplifies this approach by providing a vendor-neutral orchestration platform that integrates various tools and supports diverse workflows, as demonstrated by its clients like Gorgias and Leroy Merlin France. Such platforms aim to break down silos, enhance scalability, and maintain transparency, representing the future of operational workflows by adapting DevOps best practices across all domains.

Kestra offers a versatile platform for integrating code into workflows, providing flexibility to use any programming language, thanks to its separation of business logic from orchestration code. This approach accommodates various programming needs, such as the performance benefits of compiled languages like C or Rust and the ease of interpreted languages like Python. Kestra facilitates this integration through plugins that allow coding directly within workflows using YAML, or by executing commands for scripts stored in separate files. The platform supports dynamic and scalable workflows by enabling containerization with Docker, offering both Script and Commands tasks that can be adapted to suit different project sizes and complexities. Additionally, Kestra's flexibility allows users to seamlessly switch between programming languages without restructuring workflows, enhancing adaptability and efficiency in software development and data processing tasks.

Julien Legrand from Crédit Agricole details how the bank's data team leverages Kestra to enhance infrastructure management, improve security, and scale data pipelines across over 100 clusters for diverse operations such as NoSQL, MLOps, Streaming, and Big Data. The IT production entity, CAGIP, has transitioned from using Ansible and Jenkins to Kestra, allowing for better parallel processing, resource optimization, and security enhancements like key rotation. After initial experimentation, the team successfully integrated Kestra into their CI/CD workflows, using it for tasks like storing secrets with Vault and managing Git repositories for production flows. They have also linked Kestra to their alerting service for real-time issue notifications. Future plans include using Kestra to replace Kubernetes cron-jobs for data billing, run daily and weekly jobs for platform maintenance, and centralize backup job reports, with ambitions to tackle more complex tasks like certificate management and event-driven processes.

Kestra offers a seamless integration for orchestrating Apache Airflow jobs through its platform, allowing organizations to transition workflows without the need for complex migrations. By utilizing the Strangler Fig Pattern, Kestra enables users to gradually replace Airflow by managing both existing and new workflows, providing a centralized control plane for enhanced monitoring and orchestration. The platform simplifies the orchestration process with a declarative syntax, eliminating the need for intricate Python scripts commonly associated with Airflow's DAGs. Kestra's Airflow plugin allows for real-time visibility and tracking of task execution, providing a unified interface for managing diverse tasks across various platforms. The integration supports a gradual migration strategy that minimizes risk and leverages Kestra's advanced orchestration features, while the development team continues to expand tools and documentation to facilitate smoother transitions.

In this blog post, readers are introduced to creating serverless data pipelines using Kestra, Modal, dbt, and BigQuery, with a focus on the benefits of using Kestra for orchestration. The post outlines a project simulating an e-commerce company's sales forecasting for the holiday season, demonstrating how Kestra orchestrates the ingestion of data from an HTTP REST API into BigQuery, transformation with dbt, and running a time-series forecasting model on Modal. Kestra's orchestration capabilities are highlighted, offering structured and modular workflows, event triggers, and conditional inputs that allow for dynamic resource allocation and interactive workflow adjustments without redeploying code. Additionally, Kestra's ability to store and manage state is emphasized, providing users with the tools to monitor, troubleshoot, and optimize their workflows while leveraging serverless infrastructure for scalability and flexibility. The post underscores the value of Kestra's built-in plugins and user-friendly UI, which facilitate easy setup and management of serverless workflows, making it a compelling choice for future-proofing data platforms.

The blog post explores the distinctions between traditional CI/CD tools and the orchestration platform Kestra, emphasizing when to choose one over the other. CI/CD tools like GitHub Actions, GitLab CI/CD, Azure DevOps, CircleCI, and Jenkins are designed to automate code integration, testing, and deployment processes, each offering unique features tailored to specific environments and needs. However, Kestra stands out as an orchestration solution capable of handling complex workflows across multiple systems, providing real-time event triggers, a visual workflow editor, and robust error handling. It is particularly suitable for projects requiring extensive coordination across diverse platforms or event-driven processes. The article delves into a detailed comparison between Kestra and Jenkins, highlighting Kestra's advantages in ease of use, scalability, and its YAML-based syntax, which simplifies workflow management compared to Jenkins' Groovy scripting. Ultimately, Kestra is recommended for scenarios where traditional CI/CD tools might fall short, offering a more comprehensive solution for modern, complex applications.

Kestra provides a streamlined approach to integrating AI-powered features into workflows by connecting with Hugging Face's extensive library of pre-trained models through simple HTTP tasks. This integration allows users to perform various tasks such as sentiment analysis, language translation, and real-time data analytics without the need for complex setups. By leveraging Kestra's event-driven architecture, developers can trigger Hugging Face models in response to specific events, enabling efficient categorization and automated responses in customer support scenarios. The platform's flexibility and ease of use reduce the complexity of managing AI integrations, allowing developers to focus on creating impactful solutions with minimal maintenance. Additionally, Kestra supports modern infrastructure setups and offers a wide range of plugins, making it a versatile tool for enhancing workflows with AI capabilities.

Deploying Kestra on Clever Cloud involves utilizing Clever Cloud's platform services such as managed S3 compatible object storage, PostgreSQL databases, and Docker applications. The deployment begins by setting up a Git repository with a Dockerfile for Kestra, which is published as a Docker container. Users need to configure environment variables carefully to ensure compatibility between Clever Cloud and Kestra, including setting up a health check path and configuring PostgreSQL and MinIO storage using environment variables injected by Clever Cloud add-ons. Once the application is set up, it can be linked to Clever Cloud's Cellar S3 storage and PostgreSQL database services, which automatically inject necessary connection credentials. The deployment is completed by starting the Kestra application on Clever Cloud, where the injection of environment variables and viewing of deployment logs can be verified. Upon successful deployment, Kestra will be accessible through a specified URL, with options for user authentication to ensure security.

Kestra offers a comprehensive solution for scaling and automating dbt workflows by integrating features like Git sync, task runners, and a built-in code editor. By allowing users to sync dbt projects directly from Git repositories, Kestra ensures real-time code updates and enables seamless code editing within its platform. The use of task runners allows dynamic resource allocation, optimizing performance for complex data workflows without over-provisioning, while Docker and Kubernetes provide isolated and scalable environments for dbt tasks. Additionally, Kestra's namespace file management facilitates flexible code sharing and updates across projects, ensuring consistency between development and production environments. Enhanced logging features support the management of extensive log data, aiding in the quick identification of issues in large projects. Overall, Kestra provides a robust platform for orchestrating and scaling dbt projects, catering to the needs of both data and analytics engineers.

Introduced in Kestra version 0.19, the conditional inputs feature enhances workflow flexibility by enabling inputs to adapt dynamically based on previous user selections, thus allowing more intelligent workflow management. This feature is particularly beneficial in scenarios such as cloud resource provisioning, where selecting a provider like AWS, GCP, or Azure dynamically reveals the relevant services offered by each cloud provider. Beyond cloud orchestration, conditional inputs can improve workflows in various applications, such as access control and approval processes, by displaying options relevant to the user's role or approval level. Kestra manages these dynamic inputs using a JSON schema with the oneOf construct, ensuring that only pertinent options are shown, thereby making workflows more responsive and user-friendly. This adaptability is crucial in environments where configurations depend heavily on interdependent options, providing developers with the flexibility, efficiency, and scalability needed to manage complex workflows effectively.

Kestra 0.19.0 introduces several significant updates to enhance its orchestration platform, including a redesigned dashboard for better visualization of platform health, a new feature for Conditional Inputs to create dynamic workflows, and enhanced localization with support for 12 languages. The release also includes a Backup & Restore feature for simpler data management, system flows for automating maintenance tasks, and versioned in-app documentation. Additionally, the update improves log navigation with a new log level display, introduces a Worker Groups UI page for better task management, and provides Managed Roles for streamlined permission handling. There are also enhancements in plugin functionality, such as the ability to orchestrate Airflow DAGs and manage Azure Data Factory pipelines, along with the introduction of various new plugins. The update supports advanced scheduling features, like scheduling flows for future execution, and offers a new Concurrency Flow Tab to monitor flow concurrency. Enterprise Edition users benefit from added security features, such as encryption key setup for refresh tokens and a forgotten password reset feature. Overall, Kestra 0.19.0 aims to provide a more dynamic, accessible, and secure orchestration experience, with contributions from an expanding community of developers.