July 2023 Summaries

This blog post provides a detailed guide on setting up and running an ETL pipeline using Astro and CrateDB Cloud, emphasizing the ease and efficiency of conducting data operations in the cloud without the need for a credit card. It highlights the integration of Apache Airflow for orchestrating ETL/ELT processes and showcases CrateDB Cloud's capabilities as a distributed SQL database, offering scalability, real-time analytics, and a hassle-free managed service. The guide walks through the steps to dynamically ingest, transform, and load data from CSV files into CrateDB Cloud, performing data quality checks and executing queries. It also underlines the use of Airflow's advanced features like dynamic task mapping and the TaskFlow API for streamlined workflow management. By leveraging Astro's automated scaling and CrateDB's robust performance, the setup allows users to efficiently process large data volumes, demonstrating how to deploy and monitor these operations via Astro's cloud platform and CrateDB's admin interface.

Cosmos 1.0 is introduced as a powerful solution for running dbt Core in Airflow, offering a seamless integration that enhances data pipeline efficiency and visibility. While Apache Airflow is renowned for orchestrating data pipelines, Cosmos bridges the gap by enabling detailed monitoring and automation of dbt transformations within Airflow. Previously, integrating dbt with Airflow was cumbersome, often requiring manual interventions and dependency management, but Cosmos simplifies this by allowing users to render dbt projects as individual tasks, manage connections, and handle dependencies effortlessly. The tool is designed to be user-friendly and flexible, supporting advanced configurations and execution in Python virtual environments, thus resolving common compatibility issues. Cosmos also excels when used with Astro, further enhancing functionality with features like dbt OpenLineage integration and efficient deployment mechanisms. With its stable release, Cosmos promises to transform the way dbt and Airflow are used together, backed by strong community support and continuous development.

The Astronomer team has learned several key lessons in building their developer documentation, including the importance of atomizing documents to make them more navigable, creating a tight feedback loop between docs and product, making it easy for all team members to contribute to docs, using tools that can mature with the docs, leveraging data to prioritize work, and investing in taxonomy over process changes.

Apache Airflow is a widely-used open-source platform for authoring, scheduling, and monitoring workflows, particularly for complex data pipelines. The Airflow UI initially offers a straightforward interface with grid and graph views to monitor DAGs (Directed Acyclic Graphs) and tasks, but as systems scale, it may become insufficient for effectively managing errors and SLA (Service Level Agreement) breaches. Notifications for task successes, failures, retries, or SLA misses can be integrated with platforms like email, Slack, or PagerDuty, though SLA alerts are limited to scheduled DAGs. For more comprehensive monitoring, organizations may employ custom dashboards or integrate with third-party observability tools like Grafana, though these require significant expertise and maintenance. Alternatively, the Airflow REST API can be used to export performance metrics for visualization with BI tools such as Looker or Tableau. Astro offers a managed Airflow experience, providing out-of-the-box observability and simplified management across multi-cloud environments, making it easier for teams to monitor and optimize their workflows.

The text discusses the advantages of opting for a hosted Apache Airflow solution over self-hosting for managing ETL workflows in data engineering. Hosted Airflow providers offer simplified infrastructure management, eliminating the need for in-house setup and maintenance, which allows teams to concentrate on developing data pipelines. They also provide scalability and elasticity, enabling dynamic resource adjustments to meet workload demands without manual configuration. Additionally, hosted solutions ensure high availability and fault tolerance through built-in redundancy and disaster recovery mechanisms, reducing downtime risks. Managed updates and maintenance are another benefit, as providers handle software updates and security patches, guaranteeing access to the latest features. Lastly, hosted Airflow services offer dedicated support and expertise, which can significantly enhance troubleshooting and optimization efforts, allowing data engineers to focus on extracting insights from data without the complexities of infrastructure management.

Apache Airflow has become a fundamental tool for data engineering, evolving over nine years to integrate into the core operations of many large companies. As organizations have expanded their use of Airflow, they've faced decisions on how to architect it effectively—choosing between monolithic environments or multiple Airflows across various teams. The latter approach is seen as more scalable and adaptable to different teams' needs, as it allows for tailored environments that match specific computational requirements and upgrade schedules. The text highlights the importance of considering both horizontal and vertical scaling, as well as maintaining strong software development lifecycle (SDLC) practices for data pipelines. It also emphasizes the need for data platform teams to provide interfaces that enhance user productivity and accommodate diverse use cases, leveraging Airflow's extensive features and community tools. Overall, the focus is on fostering an environment where teams have independent access to Airflow's capabilities while ensuring reliability and scalability.

Astronomer and Fivetran have partnered to introduce a new production-grade, asynchronous ELT provider for Apache Airflow, enhancing the integration of Fivetran data ingestion with Airflow's orchestration capabilities. This new Fivetran provider replaces the legacy version, simplifying the process by combining the previous FivetranOperator and FivetranSensor into a single FivetranAsyncOperator, thus improving efficiency and resource management. The new provider leverages modern Airflow features like data-driven scheduling and asynchronous task execution to streamline workflows, allowing for more effective management of data ingestion and transformation processes. Furthermore, it integrates with OpenLineage to automatically extract and track data lineage, providing comprehensive insights into data pipelines. This collaboration underscores a commitment to offer robust support for data teams using Fivetran and Airflow, facilitating easier migration and improved functionality for orchestrating data workflows.

Astronomer is implementing policy changes that involve updating both the cross-account role policy and the operational boundary for service roles to accommodate a new Karpenter-based node autoscaling controller. These updates are necessary for Astronomer to manage and create resources essential for the controller, such as SQS queues and EventBridge rule resources, and to perform maintenance tasks. The policy adjustments include expanding permissions to enhance Data Plane cluster management, allowing the automation and support team to better address Istio ingress and RDS performance issues. Consequently, new permissions are being added to the cross-account role to support these functionalities, enhancing the reliability and support of the system.