September 2024 Summaries

Airflow has become the standard in data orchestration as companies expand their data initiatives. However, successful teams often segregate different teams into separate Airflow environments due to limitations in multi-tenancy features. These limitations include issues with execution control vs access control, implications from noisy neighbors, and user productivity. Running multiple Airflows can be a full-time job, but solutions like Astronomer's Astro provide isolated environments with complete observability, ephemeral and hibernating environments, dbt deploys, and rollback capabilities for easy upgrades and safety.

Cosmos 1.6.0 has been released, introducing new features and enhancements for the community. The update allows users to integrate dbt-core jobs into Airflow by automatically creating Airflow tasks from dbt models. Key additions include a new way of rendering source nodes as Airflow tasks through the RenderConfig API, persisting the virtual environment directory for LoadMode.VIRTUALENV, and better support for using remote manifest files in GCS/S3/Azure Blob Storage. Additionally, Cosmos 1.6 supports caching dbt package-lock.yml file to ensure predictable installation of project dependencies and enhance performance.

The Astro Terraform Provider is now generally available for public use, allowing users to automate, scale, and manage their Astro infrastructure using Terraform alongside Astronomer's API, CLI, and UI. This integration enables teams to reduce manual effort, minimize errors, and create a more streamlined approach to managing data orchestration. The provider offers features such as data sources for fetching existing information, resources for creating, managing, and destroying infrastructure components with precision, and a Terraform Import Script for seamless migration of existing resources. Astronomer is committed to driving innovation and growing its Terraform support through ongoing customer feedback, new API feature integration, open-source contributions, and enhanced migration support.

The 2024 Gartner Market Guide for DataOps Tools provides crucial insights into the emerging market of Data Operations (DataOps). This guide helps data and analytics leaders make informed decisions by offering an overview of key trends, vendor offerings, and strategic recommendations. DataOps is a collaborative practice that aims to improve communication, integration, and automation of data flows across an organization. The Gartner Market Guide identifies several must-have capabilities for DataOps tools, including data pipeline orchestration and monitoring, data pipeline observability, environment management, data pipeline test automation, and deployment automation and CI/CD. The guide also recognizes Astronomer as a Representative Vendor in the DataOps space for its Astro platform.

The text discusses the importance of ETL (Extract, Transform, Load) operations in data engineering and highlights a common pattern involving moving data from an object storage like S3 to a data warehouse like Snowflake. It provides examples of use cases for this pattern, such as ingesting unstructured data delivered by external applications or performing complex transformations using Snowflake's compute power. The text also offers a tutorial on creating a best-practice daily ingestion pipeline to move data from S3 into Snowflake using Astro Free Trial and Apache Airflow, with only basic knowledge of Airflow and Python required.

Astro Observe is a new tool that brings robust data observability capabilities to users on Astro, OSS Airflow, Amazon Managed Workflows for Apache Airflow (MWAA), and Google Cloud Composer (GCC). It offers a centralized dashboard for managing data products and a dependency graph providing a complete view of up-and-downstream dependencies within the data supply chain. The Data Products Dashboard enables teams to assign ownership, manage, and monitor their data products, while Dependency Graphs offer a complete view of lineage and ownership of dependencies across the entire supply chain for a data product. Astro Observe is currently in private preview with select design partners, moving towards general availability in early 2025.

The Airflow Summit 2024 is set to kick off in a few days, celebrating a decade of innovation and discussing upcoming advancements for Apache Airflow, including what's in store for version 3.0. Astronomer, the platinum sponsor, will showcase its fully managed Airflow service, Astro, at their booth along with new products that change the game in managing data pipelines. Expert insights and discussions on various topics related to Airflow will be led by a team of committees and PMC members. Notable speakers from companies such as Ford, Autodesk, AngelList, and King will share how they have maximized the potential of Airflow with Astronomer across diverse use cases. The event also includes an exclusive rooftop After Party for networking and relaxation under the stars.

Laurel is an AI Timekeeping platform that automates time tracking for professionals, such as lawyers and accountants. The company uses machine learning models to transform digital activities into billable entries. Airflow, a workflow management tool, plays a crucial role in orchestrating the training and deployment of these models. It helps streamline the process by serving as a unified orchestration layer while enabling dynamic model retriggering based on data policies. Additionally, it assists in managing costs associated with large language models (LLMs) used for generating time entry summaries. Overall, Laurel's strategy leverages Airflow to increase experimentation, enable model personalization, and provide scalable and cost-effective model inferences.