June 2026 Summaries

Otto, a tool introduced by Astronomer, is transforming data engineering by accelerating pipeline development and aiding in the code review process, particularly within Apache Airflow environments. By using natural language descriptions, Otto enables teams to swiftly create production-ready pipelines, debug failures, and conduct in-depth investigations without leaving their terminal. The introduction of automated DAG (Directed Acyclic Graph) reviews in platforms like GitHub and GitLab allows Otto to analyze code changes and suggest improvements before human review, addressing common Airflow-specific issues that may otherwise be overlooked. Otto leverages a comprehensive knowledge base that includes Astronomer's best practices, version-specific insights, and community standards, alongside team-specific conventions stored in Otto Memory, to provide thorough evaluations. This tool is integrated into existing CI/CD workflows and is designed to catch complex issues that could disrupt pipeline functionality. Otto's capabilities extend beyond syntax checks to a deeper understanding of operational patterns and failures, with future enhancements planned to support automated code migrations for Airflow upgrades.

Pipeline failures are common for data teams, often leading to missed service level agreements (SLAs) and delayed reports. Astronomer has introduced Otto, a data engineering agent designed to expedite identifying and resolving these failures by conducting automatic, event-driven investigations when a critical Directed Acyclic Graph (Dag) fails. Otto operates at the orchestration layer, leveraging data from Airflow such as task logs and execution context to provide immediate diagnosis, reducing Mean Time to Recovery (MTTR) from hours to minutes. This agent not only identifies whether failures stem from data, infrastructure, or code issues but also provides a structured diagnosis covering root causes, severity, and suggested fixes. Otto's integration into existing incident management workflows and its ability to conduct investigations programmatically facilitates the creation of self-healing pipelines. Moreover, Otto democratizes pipeline expertise, allowing teams to efficiently handle failures regardless of who is on call, by offering comprehensive diagnostics that replace ad-hoc investigations and reduce time spent on troubleshooting.