January 2026 Summaries

Airflow 3, released in April 2025, offers significant architectural improvements over its predecessor, including a modern React-based UI, a more efficient FastAPI-based REST API, a new Task SDK, and data-aware scheduling capabilities. However, the transition from Airflow 2 to Airflow 3 is not straightforward, as it involves breaking changes that may require substantial refactoring and rewriting of existing DAG libraries, making it a significant engineering challenge. Teams must consider whether these updates address their specific pain points and if they are willing to undertake the migration effort. Moreover, if Airflow's Python-centric nature and other limitations present ongoing challenges, teams might explore alternatives like Kestra, an event-driven orchestration platform that supports language-agnostic execution and event-driven workflows, offering a more flexible approach to orchestration. The decision to upgrade or switch should be made with consideration for future orchestration needs, especially given Airflow 2's limited support ending in April 2026.

Airflow 3 introduces significant architectural changes and improvements, such as a modern React-based UI, a new Task SDK, DAG versioning, a rebuilt FastAPI-based REST API, and data-aware scheduling, which can greatly enhance the user experience for teams deeply invested in the Airflow ecosystem. However, the migration from Airflow 2.x, which will only receive security patches after April 2026, involves substantial refactoring due to breaking changes like the removal of direct database access, deprecated features, and the need for updated deployment configurations. While these changes may benefit Python-native teams without major architectural constraints, others might face increased operational complexity and persistent limitations, such as the Python-only workflow definitions and batch-first architecture. For teams facing these issues or considering alternative orchestration models, options like Kestra offer a language-agnostic, event-driven approach with declarative YAML-based workflows, built-in governance, and the ability to integrate various coding languages without wrapper code. As organizations weigh the upgrade against their future orchestration needs, the decision to adopt Airflow 3 or explore alternatives like Kestra will depend on specific team dynamics, existing investments, and roadmap requirements.

Kestra 1.2 introduces a feature called Assets, which provides a comprehensive approach to tracking both data artifacts and infrastructure resources within workflows, offering a complete pipeline lineage. This feature allows users to declare what each task in a workflow reads and writes, encompassing data assets like tables and datasets, as well as infrastructure elements such as cloud buckets and virtual machines. By automatically generating a dependency graph, Kestra enables users to trace the full chain of dependencies, thus enhancing visibility into the entire data and infrastructure ecosystem. This capability addresses the limitations of traditional orchestration tools that primarily focus on data assets, offering a more integrated view that includes infrastructure components. Kestra's Assets are particularly beneficial for data teams managing multi-layer data pipelines, infrastructure provisioning, and operations, as they facilitate asset-based targeting and integration with external lineage systems like OpenLineage. Available in Kestra Enterprise Edition 1.2 and above, the Assets feature supports efficient collaboration and change management by maintaining an up-to-date inventory of all pipeline components, thereby simplifying the tracing of impacts and dependencies across systems.

Kestra's Assets feature enhances infrastructure management by transforming traditional "fire-and-forget" task execution into a governed and traceable catalog of live infrastructure components. This approach involves defining workflows that build a dependency graph, linking initial resource provisioning with ongoing operations, creating a comprehensive and always-updated inventory of resources complete with lineage and metadata tracking. The guide outlines how to manage Windows VMs on GCP using Kestra by registering assets like VMs, IPs, users, licenses, and snapshots into Kestra's Asset Inventory, ensuring full traceability and auditability. By treating infrastructure components as stateful assets, teams can dynamically query the asset inventory for maintenance, perform impact analysis, and maintain audit readiness, turning Kestra into a real-time configuration management database (CMDB). This unified approach ensures that every change is documented, providing teams with a complete picture of their infrastructure, facilitating better governance, auditing, and automation.

Kestra Assets revolutionizes data and infrastructure governance by integrating it directly into workflow orchestration, offering a seamless approach to tracking, cataloging, and lineage mapping across various resources. It treats tables, files, and cloud assets as first-class entities, linking them to specific workflows through declarative asset declarations that capture inputs and outputs. This system is technology-agnostic, supporting diverse tools and languages, thus ensuring consistent governance regardless of the underlying technology. By automatically generating lineage graphs and maintaining execution histories, Kestra Assets provides real-time visibility into data flows and infrastructure dependencies, reducing manual documentation and guesswork. This enhances collaboration across data, platform, and engineering teams by making assets discoverable and reusable, while also enabling impact analysis, clear resource ownership, and enforceable modeling conventions. The innovation lies in transforming orchestration from mere task execution to a comprehensive platform that understands and governs the entire data ecosystem, ensuring transparency, trust, and accountability in data operations.

Kestra 1.2 introduces significant performance upgrades aimed at reducing execution latency and database contention, particularly under large workloads with parallel workflows. The release features a new parallel file loading strategy, which enables faster task startup by downloading namespace files concurrently using multiple CPU cores, significantly reducing startup times. Additionally, the update enhances concurrency performance by minimizing database locking through grouped execution message processing, resulting in a twofold improvement in scenarios with high concurrency, such as ForEach tasks. These optimizations contribute to a more scalable and predictable performance without requiring workflow changes, underscoring Kestra's commitment to efficient large-scale orchestration.

As data orchestration needs in 2026 evolve, companies are reevaluating their reliance on Apache Airflow, which, while initially successful for Airbnb's scheduling issues, falls short on modern enterprise requirements like multi-tenant governance, real-time processing, and hybrid deployments. Several alternatives are available, each catering to specific needs: Kestra offers a versatile, language-agnostic approach allowing orchestration across various languages and environments; Astronomer provides managed Airflow services for those committed to the platform; Dagster emphasizes data asset orchestration; Prefect simplifies Python-centric workflows; and AWS Step Functions and Azure Data Factory cater to cloud-native ecosystems. Kestra stands out with its ability to manage diverse workloads without pushing for extensive rewrites, offering deployment flexibility and governance features like RBAC and namespace isolation. As enterprises face the inevitability of technological change, choosing an orchestration tool that supports future growth and diversification without locking into a single language or paradigm becomes critical.

Apache Airflow, widely recognized for orchestrating data pipelines, faces challenges in meeting enterprise requirements like multi-tenant governance and real-time event processing, which were not part of its original design when it was developed at Airbnb in 2014. As of 2026, with Airflow 2 reaching end-of-life, organizations are evaluating alternatives that offer more flexibility and modern features to address governance, deployment, and language lock-in issues. Options include Kestra, which is language-agnostic and supports various deployment scenarios, and other tools like Astronomer for managed Airflow, Dagster for asset-centric orchestration, and Prefect for lightweight Python workflows. Enterprises are advised to consider their current and future orchestration needs, especially as workflows increasingly overlap across data engineering, machine learning, and application development, making Kestra's universal approach appealing for those seeking to avoid language and paradigm lock-in.

Flaky tests, which can unpredictably pass or fail without code changes, pose a significant challenge in software development due to factors like timing and resource contention. At Kestra, where over 6,000 tests run across repositories, these issues became pronounced, prompting a journey to address them. Initially, retrying tests using JUnit's retry annotation was attempted, but this only partially alleviated the problem, as it inflated test times and masked underlying issues. The team then focused on fixing tests by improving resource management through custom JUnit extensions, yet some tests remained problematic or had to be disabled. Ultimately, Kestra embraced the inevitability of some tests failing by flagging them as flaky, allowing them to fail in CI without affecting the overall result. This pragmatic approach separates critical tests from those allowed to fail, ensuring reliable CI signals while maintaining test coverage and addressing the challenges of non-determinism.

Kestra 1.2 introduces several significant enhancements, including enterprise-grade Asset management that maintains a stateful inventory of external resources and establishes dependency relationships within workflows, facilitating data pipeline orchestration with automatic lineage tracking. The release also adds Templated Custom Blueprints, enabling users to create reusable, configurable workflows without needing to modify YAML, and offers comprehensive revision history for Namespace Files to track and restore previous versions. Enhancements also include support for concurrent executions from any trigger type, a redesigned Execution Overview page for improved user experience, and new plugins like Segment CDP, AWS EMR Serverless, and Pipedrive. Additionally, Kestra 1.2 improves secret management integrations and facilitates the deployment of Kestra on AWS and Azure Marketplaces, providing a production-ready orchestration platform with easier setup and management.

Kestra operates a comprehensive data stack managed by a single analytics engineer, leveraging tools like PyAirbyte for data ingestion, dbt for transformations, and Lightdash for visualization, all orchestrated through Kestra's unique framework that allows existing scripts and tools to be used without modification. This setup supports various teams including sales, marketing, product, and leadership by providing real-time insights and facilitating data-driven decision-making. Kestra's architecture, which separates orchestration logic from application code, allows for flexible and incremental evolution of data pipelines, accommodating various programming languages and tools. The use of declarative YAML for workflow definitions and the deployment of flows via Terraform ensures that changes are tested before reaching production, maintaining operational stability. The orchestration model supports subflows for reusability and namespaces for smart alerting, enabling scalable and consistent pipeline management without duplicating configurations. This streamlined approach empowers the analytics engineer to focus on business insights and operational efficiency, rather than on reconciling different technologies, underscoring Kestra's adaptability to the complexities of modern data environments.