May 2026 Summaries

The process of releasing software updates was initially manual and time-consuming, requiring frequent attention and coordination between GitHub repositories, especially on Tuesdays when multiple versions were released simultaneously. To streamline this process, the author utilized Kestra, an orchestration software, to automate and simplify the release workflow. By developing a series of flows using YAML, the release process was transformed from a manual operation to an automated one, allowing for parallel processing of open-source and enterprise editions, incorporating retry logic for flaky tests, and eventually automating Docker image publication. This automation significantly reduced the need for constant monitoring and manual intervention, allowing team members to focus on other tasks and improve efficiency. The adoption of this system was seamless within the team, as the YAML configuration was easy to understand and modify, enabling the creation of additional flows for other routine tasks without the need for formal training or meetings.

The decision between using dbt Core or dbt Cloud revolves around how much operational complexity a team is willing to manage and the specific needs of their data workflows. dbt Core, being an open-source, command-line tool, offers maximum control and flexibility, making it suitable for engineering-heavy teams who can handle infrastructure management and prefer to integrate dbt into existing systems. Conversely, dbt Cloud is a managed SaaS platform that simplifies operations by providing a web-based development environment, built-in scheduling, and collaboration tools, which is ideal for teams seeking to optimize for speed and reduce operational overhead. Cost considerations play a significant role, as dbt Core requires investment in infrastructure and engineering time, while dbt Cloud involves subscription fees offset by reduced infrastructure costs. Ultimately, the choice depends on factors like team size, technical expertise, project complexity, budget, and how dbt integrates with the existing data stack, with platforms like Kestra enhancing orchestration for both options.

The Kestra MCP server has expanded its capabilities with three new documentation tools—search_docs, get_doc, and list_doc_children—that enhance AI coding agents' ability to reason about the platform, beyond just writing valid flows. These tools, integrated seamlessly with existing systems like Claude Code and other MCP-compatible agents, allow for more comprehensive understanding and navigation of Kestra's documentation by enabling keyword searches, retrieval of full markdown content, and structured navigation through documentation pages. This development elevates AI agents from merely being flow writers to Kestra architects, enabling them to make informed platform decisions regarding storage backends, RBAC structures, and task runner configurations, thereby turning data workflows into opportunities for strategic insights and optimizations.

Airflow 2 has reached its end of life, prompting many teams to consider upgrading to Airflow 3, although this transition poses challenges due to the complexity of migration and the opportunity to evaluate alternative solutions like Kestra. The author suggests that the traditional process of upgrading often fails because teams are hesitant to deviate from the status quo without a working example to justify the shift. However, the introduction of large language models (LLMs) and tools like migration-skills have streamlined this process, allowing for a quicker conversion of Airflow DAGs to Kestra flows, thus facilitating easier demonstration and evaluation of Kestra's benefits. Kestra offers a different approach with declarative YAML flows, file-based data passing without size limits, and a rich plugin ecosystem that enhances integration capabilities, supported by an IDE-like UI that improves workflow management. The migration process involves using AI coding agents to convert Python DAGs into Kestra YAML flows, which are validated against live schema to ensure accuracy, and deploying the resulting configurations using the Kestra CLI, allowing for a structured transition from Airflow to Kestra with improved data handling and execution visibility.

In an examination of AI workflow orchestration, the author details a Kestra-powered system that automates the routing of GitHub issues to the correct product squad using an internal ownership map from Notion. This system efficiently handles high demand and potential failures of AI models like Gemini 3.1 Pro by implementing fallbacks to older versions and utilizing deterministic steps for most tasks, aside from the AI classification step. The Kestra flow incorporates practices such as retry policies to manage transient failures, concurrency limits to prevent exceeding quotas, and a unified alert path to maintain operational consistency. The author argues that treating AI models as one component within a larger workflow, rather than the focal point, allows for more manageable and reliable operations, highlighting the importance of orchestration in smoothing over the complexities and potential disruptions in AI workflows.

Workflow orchestration is crucial for coordinating complex tasks across data pipelines, infrastructure jobs, and business processes, addressing the challenge of coordination beyond mere execution. The Kestra Fundamentals course offers a structured approach to mastering this discipline through four modules, providing hands-on examples and culminating in a certification exam. The course emphasizes understanding the distinction between orchestration and scheduling, highlighting Kestra's language-agnostic nature, where workflows are written in YAML, and tasks can run in any language, facilitating seamless integration with existing systems. Participants learn to effectively manage multi-step workflows, handle errors, and use plugins and blueprints to connect with over 1200 external systems, such as PostgreSQL and Slack, allowing them to adapt and create production-ready workflows. By completing this course, participants not only gain a deeper understanding of orchestration concepts but also earn a certification that validates their ability to design and implement efficient workflows, which can be showcased on professional platforms like LinkedIn.