September 2026 Summaries

Astronomer has expanded its Airflow development tools across browsers, local editors, and agent-driven workflows, highlighted by the general availability of the browser-based Astro IDE, a new VS Code extension, and planned Astro CLI improvements. The Astro IDE enables users to import Git projects, use the Otto assistant to draft or modify code, test DAGs against deployments, and deploy or commit changes, with Astronomer reporting faster onboarding and up to tenfold quicker DAG delivery for adopting teams. While Blueprint, its visual DAG builder, and Otto remain in Labs, the IDE has already been used to create nearly 7,000 projects across about 2,300 organizations. The VS Code extension, with a Cursor version planned, brings deployment health monitoring, error investigation, Otto integration, local Airflow operation, hibernated deployment controls, Airflow upgrade guidance, and embedded documentation directly into editors. Astronomer is also redesigning the Astro CLI around a consolidated pyproject.toml configuration and features intended to better support both data engineers and coding agents.

Astronomer and Carnegie Mellon’s Database Group created de-bench, a 139-task benchmark intended to evaluate AI agents on realistic production data-engineering environments centered on Airflow, addressing a gap left by benchmarks focused primarily on SQL, dbt, or analytics. The benchmark simulates a multi-team platform with 114 DAGs, about 650 Airflow tasks, dbt projects, legacy schedulers, governance documents, incomplete migrations, and tasks involving pipeline failures, data-quality investigations, documentation, and Airflow upgrades. In tests across Anthropic and OpenAI models, Astronomer reports that its Otto agent harness generally achieved higher accuracy and reliability than Claude Code and Codex when using comparable models and reasoning levels, while often costing less; model selection had the largest overall effect on performance, while additional reasoning produced diminishing returns at higher levels. Otto’s largest advantage appeared in Airflow upgrade and version-compatibility tasks, where its access to curated, current, version-specific Airflow knowledge helped it resolve removed imports, renamed APIs, and provider conflicts that models could not reliably infer. To support trustworthy measurement, de-bench uses known-good solutions, failing baseline projects, executable checks for code tasks, repeated trials, replay tests for idempotency, recomputation of expected data outputs, and manually audited model-based grading for written investigations.

Data teams often struggle to create warehouse columns that require subjective judgments, such as classifying leads, detecting spam, or resolving whether records refer to the same entity, because SQL rules are brittle, traditional machine learning requires specialized resources, and general-purpose LLMs can be costly, slow, and inconsistent in their outputs. The post argues that TypeSafe’s new Jev model offers a potential alternative by returning strictly typed classifications, scores, or Boolean values with confidence scores rather than free-form text, making it better suited to production data pipelines. In an informal test classifying job titles, Jev was substantially faster and cheaper than Snowflake Cortex AI_CLASSIFY, while its confidence scores appeared useful for distinguishing reliable decisions from ambiguous ones. The author proposes that these “semantic columns” could enable teams to apply judgment across entire datasets rather than samples, using confidence-based workflows that automatically accept high-confidence results, escalate uncertain cases to larger models, and send difficult cases to human review. Potential applications include lead scoring, entity resolution, document evaluation, ticket routing, and intelligently classifying task failures for orchestration retries. Although the author emphasizes that typed outputs and confidence scores do not guarantee correctness and require testing against labeled data, they predict that inexpensive specialized judgment models will make many previously impractical data-enrichment tasks commonplace, with Airflow integrations intended to support their deployment and review workflows.

Apache Airflow 3.3 introduces intra-task durable execution, allowing long-running tasks to recover from worker crashes without repeating completed work, reducing duplicate external jobs, unnecessary compute costs, and repeated AI model or tool calls. Through AIP-103, Airflow adds task and asset state stores: task-scoped persistent key-value storage for retries within a DAG run, useful for job IDs, offsets, and checkpoint pointers, and asset-scoped storage for cross-run state such as watermarks. Existing operators including Spark, Kubernetes, BigQuery, Databricks, Snowflake, Redshift, Glue, and others can now persist external job identities and reattach on retry by default, while AI agent tasks can opt into step-level caching with `durable=True` to replay completed model and tool calls. The approach differs from replay-based workflow engines because Airflow focuses on reconnecting to external systems or resuming from explicitly recorded checkpoints rather than requiring deterministic workflow code. Durable execution provides at-least-once recovery rather than exactly-once guarantees, so idempotency, stable identifiers, and status checks remain necessary for side effects. These capabilities support data engineering, ML checkpoint recovery, and AI agent resilience while replacing common workarounds such as custom offset tables, Airflow Variables, split submit-and-wait tasks, and custom sensors.

Astronomer has expanded Otto, its AI agent for Airflow and Astro, from the Astro IDE to a context-aware panel available throughout Astro’s browser interface, enabling users to ask deployment and DAG questions, troubleshoot failures, and approve actions without repeatedly providing context. The company has also introduced a hosted, authenticated MCP server that lets MCP-compatible agents such as Claude Code, Cursor, Codex, Claude Desktop, and VS Code connect to Astro using existing credentials and permissions, providing access to Astro control-plane resources and Otto’s capabilities. For Enterprise Business Critical customers, Astronomer is offering limited early access to Bring-Your-Own-Model support for approved OpenAI- or Anthropic-compatible endpoints, while recommending Otto’s managed models for most teams due to automatic updates and cost optimization. Upcoming developments include a Carnegie Mellon partnership on agent benchmarking and extensions for VS Code and Cursor that will integrate deployment browsing, DAG controls, logs, and Otto.