Company
Date Published
Author
Nick Acosta
Word count
745
Language
English
Hacker News points
None

Summary

Fivetran has introduced an async provider for Airflow, developed in collaboration with Astronomer, allowing data engineers to run Fivetran data syncs more efficiently in Airflow 2.2+. The new provider utilizes deferrable operators and sensors, enabling asynchronous execution of tasks without consuming worker slots, thus improving the overall efficiency of Airflow deployments. This allows for I/O bound operations like data syncing to be performed without blocking other tasks or DAGs. The Fivetran async provider can be added to any Airflow 2.2+ environment with pip install airflow-provider-fivetran-async, and provides examples on how to use it in DAGs.