March 2026 Summaries

Kestra is establishing itself as a leader in the emerging infrastructure category of orchestration, which is becoming crucial for enterprises integrating AI into production. As companies face challenges not with the capabilities of AI models but with coordinating the surrounding processes, Kestra offers a unified orchestration layer that spans infrastructure, data, and AI, aiming to replace fragmented automation systems with a centralized, adaptable solution. This approach is validated by its adoption across more than 30,000 organizations, including major enterprises like JPMorgan Chase, BHP, Apple, and Toyota, which have seen improved workflow management, governance, and reduced engineering dependency. With a recent $25 million Series A funding round led by RTP Global, Kestra plans to further develop its platform to cater specifically to data, AI, and infrastructure teams, emphasizing a flexible orchestration layer that can adapt to various infrastructures without compromising control or simplicity. As the orchestration category gains traction, Kestra's open-source project continues to grow, driven by engineers who recognize its value in managing complex systems, and the company is poised to accelerate its impact globally by enhancing its platform and expanding its team.

dbt Cloud's built-in scheduler efficiently handles scheduling dbt jobs, running continuous integration (CI) checks on pull requests, and hosting a browser IDE, but it is limited to managing only the transformation layer of data pipelines. This limitation means that upstream ingestion processes and downstream activation steps must be managed separately, often requiring custom solutions such as webhooks or scripts. The scheduler also lacks cross-tool coordination and lineage tracking beyond the dbt models, leading teams to use external orchestrators like Kestra. Kestra complements dbt Cloud by orchestrating the entire data pipeline, handling triggers, lineage, and failure management across different tools and languages, thus providing a comprehensive view of the pipeline. By managing tasks in one YAML file and allowing event-driven triggers, Kestra ensures a seamless workflow from data ingestion to activation, overcoming the limitations of dbt Cloud's consumption-based pricing and isolated failure handling. This orchestration approach helps mitigate issues such as stale data and silent failures by providing robust cross-stack lineage and alerting capabilities, making it a valuable addition for teams with complex, multi-tool pipelines.

The text explores the evolving landscape of data handling through SQL and dataframes, highlighting the integration and synergy between these two methods in data analytics. Initially, SQL was primarily used by data engineers for querying large datasets in data warehouses, while dataframes were favored by data scientists for in-memory computation and data manipulation in languages like Python. However, the boundaries between these approaches are increasingly merging, with tools like Pandas, Polars, DuckDB, and others offering capabilities that blend SQL's declarative querying with dataframe's imperative transformations. Polars, for instance, is a high-performance DataFrame library that combines efficient memory usage with a SQL context, while DuckDB provides an in-process OLAP DBMS with support for both SQL and dataframe operations. The text also discusses frameworks like Modin and Fugue, which aim to scale dataframe operations across distributed systems, and highlights products like Ponder that enable execution of dataframe code in cloud environments like BigQuery. Overall, the integration of SQL and dataframes offers versatile options for data processing, enabling seamless transitions between different tools and environments based on specific needs and expertise.

VMware continues to be a vital component of enterprise infrastructures, primarily through its vSphere environments that support virtualization across various workloads. However, the automation surrounding VMware operations often relies on fragmented solutions such as scripts and legacy tools. The Kestra platform offers a modern approach by integrating VMware operations into broader orchestration workflows, allowing seamless management of virtual machine lifecycles, infrastructure automation, and application workflows. This integration enables teams to orchestrate VMware tasks alongside other operational workflows, replacing fragile scripts with robust, version-controlled automation that can respond to real-time infrastructure events. Key features include VM lifecycle management through vCenter and ESXi, automated snapshot management, template and cloning capabilities, and event-driven automation that reacts to specific VMware lifecycle events. This approach transforms VMware from an isolated system into a component of a larger orchestration model, aligning it with contemporary infrastructure automation practices and enhancing operational efficiency.

Kestra is highlighted as an ideal orchestrator for dbt Core teams due to its ability to manage tasks that dbt Core itself does not handle, such as scheduling, retry logic, and alerting. While dbt Core focuses on data transformation within data warehouses, it lacks production runtime capabilities, necessitating an orchestrator like Kestra to ensure seamless operation. Kestra integrates with both dbt Core and dbt Cloud, providing orchestration for broader data pipelines, including ingestion and activation processes. It offers a YAML-based workflow, which aligns with dbt's syntax, reducing the learning curve for teams accustomed to YAML and SQL. Kestra's features include task sequencing, retry mechanisms, failure alerting, and real-time monitoring, ensuring reliable data processing and allowing teams to track cross-pipeline lineage effectively. Additionally, Kestra supports event-driven triggers, enabling dbt tasks to run as soon as the data is ready, thus optimizing workflow efficiency.

Infrastructure automation often appears orderly on diagrams but is complex in reality, involving multiple tools and teams managing hardware, virtual environments, and networks. Kestra 1.3 addresses this complexity by providing a control plane that orchestrates processes rather than replacing existing tools, turning infrastructure operations into workflows with features like retries, timeouts, and audits. New plugins support integration with common infrastructure management tools, such as Argo CD for GitOps delivery, Cloudflare for edge operations, MAAS for bare metal management, and NetBox for maintaining source-of-truth workflows. Kestra enhances reliability and observability by integrating infrastructure automation, making handoffs between tools explicit and reliable. It supports virtualization and recovery workflows, with plugins for KVM/libvirt and Nutanix AHV that cover VM lifecycle management and snapshotting. Kestra’s Blueprints offer pre-made workflow patterns for common orchestration challenges, including GitOps and configuration drift detection. Additionally, Kestra provides an alternative to VMware’s Aria/vRA orchestration layer, allowing teams to modernize their infrastructure management without a complete overhaul of existing VMware ecosystems.

In early 2026, advancements in AI have significantly reshaped data engineering by lowering the entry barrier and expanding the role's scope. AI tools like Claude Code now enable a broader range of professionals, from analytics engineers to legal ops managers, to engage in data engineering tasks previously reserved for specialists. This democratization has led to a fragmentation of the data engineer role, with distinct positions such as platform engineers, workflow engineers, and AI engineers emerging, each contributing to the orchestration of complex systems. The challenge has shifted from writing code to coordinating workflows across diverse systems, necessitating a focus on orchestration thinking rather than traditional engineering. Declarative, language-agnostic tools like YAML are gaining popularity for their ability to separate orchestration logic from execution logic, making it easier for AI to assist in generating and maintaining workflows. The rise of AI has also prompted predictions about the future of the field, suggesting that roles like "workflow engineer" will become formalized, and new governance tools will emerge to manage the complexity of distributed workflows. As AI continues to commoditize the coding aspect of data engineering, the emphasis will increasingly be on operations engineering, focusing on reliability and incident management.

Kestra 1.3 enhances its platform by introducing production-grade controls, CI/CD tools, and infrastructure plugins that cater to enterprise, developer, and infrastructure teams. Key features include a Kill Switch for immediate intervention during production issues, centralized Credentials management to streamline server-to-server authentication, and an expanded Assets system for real-time lifecycle management and automation. AI Copilot now offers improved governance with RBAC controls and wider UI availability, while the new kestractl CLI facilitates consistent automation across environments. The release also features a new GitHub Action for seamless CI integration and a Plugin Defaults UI to simplify configuration management. These updates, along with new plugins for infrastructure and policy integration and a focus on observability and governance, reflect Kestra's commitment to enhancing operational efficiency and collaboration across different team workflows.