October 2022 Summaries

Astronomer emphasizes the importance of continuous data quality checks to ensure the reliability of data-driven insights and decisions, utilizing Apache Airflow within its data platform. The company's approach involves orchestrating data flows from ingestion to dashboards using Airflow DAGs, with a focus on timely alerts for data errors and maintaining data integrity through structured checks. Their system features a standalone data-quality DAG for high-impact pipelines and integrated checks during table creation, leveraging SQL operators to facilitate these tasks. By implementing a robust architecture, Astronomer addresses potential data issues proactively, using alerts to notify the team of data anomalies and ensuring that data quality is maintained consistently. This setup not only enhances observability and trust in dashboards and models but also allows for iterative improvements and the integration of domain knowledge into data quality processes. The company shares its journey to encourage other organizations to adopt similar practices for reliable data governance.

Astro Python SDK 1.1 introduces significant enhancements for data engineers and scientists by incorporating data-driven scheduling, dynamic tasks, and Redshift support. The update leverages Airflow 2.4's Dataset objects to enable modular DAGs that communicate through direct data dependencies, improving efficiency and maintainability. By refactoring DAGs into smaller, modular units and using Table and File classes that inherit from Dataset, data-driven scheduling is achieved, allowing DAGs to trigger based on real-time data readiness. This method eliminates traditional scheduling workarounds, enhancing collaboration and reducing code complexity. Dynamic task mapping, a feature from Airflow 2.3, is now integrated into the SDK, allowing DAGs to generate parallel tasks during runtime, which improves adaptability and runtime efficiency. Additionally, the SDK now supports Redshift, providing an optimized path for loading data from S3, in line with other data warehouses like Snowflake and Google BigQuery. Overall, these improvements aim to streamline the creation of modular, data-driven pipelines, enhancing both performance and ease of use.

StarPower, a pseudonym for Astronomer, exemplifies how a data-driven company optimally coordinates its data operations using Airflow as a central but unobtrusive orchestration tool. This setup allows teams across the organization to independently develop and deploy data pipelines with minimal training, leveraging a shared ecosystem for efficient data processing and analysis. By employing a control plane, StarPower ensures visibility and governance over its data operations, facilitating collaboration and innovation. The orchestration platform not only supports the rapid development of new data flows but also integrates tools like OpenLineage for data lineage, enhancing troubleshooting and documentation. This infrastructure reduces administrative overhead and fosters a collaborative environment, empowering non-experts to contribute effectively to data projects while maintaining high standards of quality and documentation. With these capabilities, the orchestration platform transforms Airflow from a mere job scheduler into a comprehensive system that supports the entire data lifecycle, driving critical business decisions through a well-managed network of data.

Apache Airflow® 2.4 introduces a transformative shift from monolithic data pipelines to micropipelines, utilizing Datasets as a core concept to enable more efficient, scalable, and maintainable data workflows. This change allows data pipelines to be decomposed into smaller, independent components—micropipelines—that can be triggered by dataset updates rather than time schedules. This approach resolves common issues with monolithic pipelines, such as delayed data availability and development friction, by enabling independent scaling and deployment of micropipelines that can be programmed in various languages, such as Python or SQL, to suit different tasks. The Astro Python SDK further enhances this functionality by providing an abstraction layer for Datasets, facilitating seamless data movement and integration across diverse cloud storage and database systems. This evolution supports more predictable orchestration of data products, aligning with the principles of DataMesh for decentralized data ownership and self-service data analysis, thereby accelerating the availability of business-critical insights.

Expanding data access and exchange within a company involves balancing team autonomy with effective data sharing across multiple teams. At Astronomer, a centralized ecosystem has been developed to manage data more efficiently, allowing data teams to focus on analysis rather than operational tasks. This system leverages Astro, a managed service built on Apache Airflow, to unify siloed data teams. The data science team at Astronomer, led by Steven Hillion and Taylor Merrick, has worked on creating a centralized data warehouse and technical ecosystem to support various teams in their data endeavors, from building dashboards to contributing to the company's data warehouse. The integration of data across the company presents challenges such as maintaining data pipeline operations, but the use of tools like Airflow and Astro facilitates this process. The company aims to democratize data usage by encouraging more employees to contribute to operational DAGs, using a cloud-based IDE to simplify the process, ultimately fostering an environment where data-driven decision-making is accessible to all teams.