August 2021 Summaries

Data silos, isolated data repositories controlled by individual departments, can limit an organization's ability to obtain a comprehensive view of its operations, leading to issues such as reduced data integrity, compromised security, and decreased productivity. Traditionally, data centralization—maintaining a single, unified database—has been seen as a solution, but it often results in complex, monolithic systems that can create hyper-specialized silos and inhibit flexibility. Modern approaches, like data meshes, offer a more practical solution by connecting multiple data warehouses across an organization, allowing for centralized governance while enabling distributed, interconnected data management. This method, facilitated by tools like Apache Airflow® and Astronomer, can enhance data orchestration and integration, providing a richer, more nuanced insight into organizational dynamics and opportunities.

Societe Generale, a multinational investment bank, adopted Apache Airflow® as part of their Open Banking transformation to transition from an ITIL model to a DevOps approach, emphasizing shared responsibility and developer autonomy. After evaluating various orchestration tools, they selected Apache Airflow® for its user-friendly interface, community support, and Python-based extensibility, which facilitated interactions with both private and public clouds, and supported standard ETL and ELT workflows. This shift resulted in significant license cost savings, accelerated development workflows, and the establishment of over 150 infrastructures. As a Product Owner, Alaeddine Maaoui highlighted the successful integration of open-source tools in mission-critical banking environments and emphasized the importance of selecting tools suited to specific use cases. Looking ahead, Societe Generale plans to upgrade from Airflow 1.X to 2.X with support from Astronomer, demonstrating their commitment to modernizing their data orchestration processes.

Modern data pipelines have revolutionized data handling by automating the flow of large data volumes from their sources to destinations, such as data warehouses or visualization tools, thereby enabling valuable insights and data-driven decision-making. These pipelines can range from simple data extraction and loading to complex processes involving machine learning. Components of a data pipeline include the data origin, destination, flow, storage, processing, monitoring, and alerting systems, while their types encompass batch, real-time, cloud-native, and open-source variants. Automated solutions like Airflow offer scalability, flexibility, and cost-efficiency, reducing manual efforts and errors while enhancing real-time data processing, scalability, data quality, and analytical capabilities. Organizations benefit from the reduction in operational overhead and the ability to focus on leveraging data for strategic insights. The successful implementation by Herman Miller illustrates the transformative impact of using platforms like Airflow, which streamline data operations, enhance accuracy, and facilitate easy monitoring and development, allowing businesses to focus more on data utilization than on the underlying technology.

The blog post discusses the development of a Python package named Cosmos, designed to facilitate running dbt models on Airflow by generating Airflow DAGs from dbt models more conveniently. This piece serves as the third installment in a series exploring the integration of dbt transformation pipelines within an Airflow DAG, utilizing the dbt manifest.json file to map each model to a task. The post introduces a utility, DbtDagParser, which simplifies the creation of Airflow task groups by parsing dbt models and dependencies, offering flexibility through features like the "dbt_tag" parameter for selective model execution. While the method provides fine-grained control over model execution, it may impact performance due to the overhead of calling "dbt run" for each model. Tests showed that a mapped DAG took longer to execute than a single-task DAG, highlighting tradeoffs concerning runtime and control. The post encourages users to evaluate this pattern against their needs, keeping in mind the performance implications based on their dbt model structures and data volumes, and hints at further explorations of dbt and Airflow integration in future posts.