Durable execution with Apache Airflow®

Apache Airflow 3.3 introduces intra-task durable execution, allowing long-running tasks to recover from worker crashes without repeating completed work, reducing duplicate external jobs, unnecessary compute costs, and repeated AI model or tool calls. Through AIP-103, Airflow adds task and asset state stores: task-scoped persistent key-value storage for retries within a DAG run, useful for job IDs, offsets, and checkpoint pointers, and asset-scoped storage for cross-run state such as watermarks. Existing operators including Spark, Kubernetes, BigQuery, Databricks, Snowflake, Redshift, Glue, and others can now persist external job identities and reattach on retry by default, while AI agent tasks can opt into step-level caching with `durable=True` to replay completed model and tool calls. The approach differs from replay-based workflow engines because Airflow focuses on reconnecting to external systems or resuming from explicitly recorded checkpoints rather than requiring deterministic workflow code. Durable execution provides at-least-once recovery rather than exactly-once guarantees, so idempotency, stable identifiers, and status checks remain necessary for side effects. These capabilities support data engineering, ML checkpoint recovery, and AI agent resilience while replacing common workarounds such as custom offset tables, Airflow Variables, split submit-and-wait tasks, and custom sensors.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.