Company
Date Published
Author
Viraj Parekh
Word count
1364
Language
English
Hacker News points
None

Summary

Astronomer and Fivetran have partnered to introduce a new production-grade, asynchronous ELT provider for Apache Airflow, enhancing the integration of Fivetran data ingestion with Airflow's orchestration capabilities. This new Fivetran provider replaces the legacy version, simplifying the process by combining the previous FivetranOperator and FivetranSensor into a single FivetranAsyncOperator, thus improving efficiency and resource management. The new provider leverages modern Airflow features like data-driven scheduling and asynchronous task execution to streamline workflows, allowing for more effective management of data ingestion and transformation processes. Furthermore, it integrates with OpenLineage to automatically extract and track data lineage, providing comprehensive insights into data pipelines. This collaboration underscores a commitment to offer robust support for data teams using Fivetran and Airflow, facilitating easier migration and improved functionality for orchestrating data workflows.