Content Deep Dive
Introducing the Fivetran async provider in Airflow
Blog post from Fivetran
Post Details
Company
Date Published
Author
Nick Acosta
Word Count
745
Language
English
Hacker News Points
-
Summary
Fivetran has introduced an async provider for Airflow, developed in collaboration with Astronomer, allowing data engineers to run Fivetran data syncs more efficiently in Airflow 2.2+. The new provider utilizes deferrable operators and sensors, enabling asynchronous execution of tasks without consuming worker slots, thus improving the overall efficiency of Airflow deployments. This allows for I/O bound operations like data syncing to be performed without blocking other tasks or DAGs. The Fivetran async provider can be added to any Airflow 2.2+ environment with pip install airflow-provider-fivetran-async, and provides examples on how to use it in DAGs.