Astronomer Data Engineering Benchmark

Astronomer and Carnegie Mellon’s Database Group created de-bench, a 139-task benchmark intended to evaluate AI agents on realistic production data-engineering environments centered on Airflow, addressing a gap left by benchmarks focused primarily on SQL, dbt, or analytics. The benchmark simulates a multi-team platform with 114 DAGs, about 650 Airflow tasks, dbt projects, legacy schedulers, governance documents, incomplete migrations, and tasks involving pipeline failures, data-quality investigations, documentation, and Airflow upgrades. In tests across Anthropic and OpenAI models, Astronomer reports that its Otto agent harness generally achieved higher accuracy and reliability than Claude Code and Codex when using comparable models and reasoning levels, while often costing less; model selection had the largest overall effect on performance, while additional reasoning produced diminishing returns at higher levels. Otto’s largest advantage appeared in Airflow upgrade and version-compatibility tasks, where its access to curated, current, version-specific Airflow knowledge helped it resolve removed imports, renamed APIs, and provider conflicts that models could not reliably infer. To support trustworthy measurement, de-bench uses known-good solutions, failing baseline projects, executable checks for code tasks, repeated trials, replay tests for idempotency, recomputation of expected data outputs, and manually audited model-based grading for written investigations.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.