March 2022 Summaries

Dagster has integrated with Airbyte, allowing users to orchestrate data pipelines and gain detailed visibility into each step of the process. This integration provides a live view of sync status, including raw logs from Airbyte, making it easy to track changes over time. Users can configure Airbyte resources to point to their hosting instance and use pre-built ops or software-defined assets to generate code for specific tables. The integration also enables users to visualize their data pipeline in Dagit, giving a more detailed look into the purpose of each step. This reflects Dagster's focus on positioning itself as the centralized place to discover and manage the status of an entire data platform.

Dagster version 0.14.0 introduces a more mature software-defined asset system, allowing users to explicitly declare tables, ML models, and datasets that they want to exist, with tightly linked assets to computations that generate their contents. This enables a reconciliation-based approach to data orchestration, adds a new dimension to data observability, and helps make Python a native citizen of the Modern Data Stack. The release also includes new integrations such as Dagster-Airbyte and Dagster-Pandera, improved operational maturity with a new Dagit homepage and auto-start sensors and schedules, and various other features and improvements.

The text discusses the challenges of managing data assets in modern data platforms, particularly the lack of metadata and the need for a more integrated approach to code and data. It introduces the concept of bridging the code-data gap by carving out a place in an orchestrator's data model for richly annotated assets. The text also explores the introduction of new features in Dagster 0.14.0, including an API for attaching metadata to assets, a Table Schema API for expressing tabular schemas as metadata, and an integration with Pandera, a library that implements data validation for Pandas-like dataframes. These features aim to improve observability, integrity, and maintenance of data quality in the field of data engineering.

The software-defined asset is a declarative approach to managing data and orchestrating its maintenance. It combines the description of how to compute an asset with metadata about that asset, including dependencies. This approach is particularly appealing because it makes systems more debuggable, comprehensible, and automatable by making intentions explicit and offering a principled way of managing change. The software-defined asset can be invoked by the orchestrator to materialize that asset, i.e., to run the op on the contents of the upstream assets and then persist the results in storage. A collection of software-defined assets constitutes an asset graph, which is critical for understanding and working with data. The asset graph offers a cross-technology way of answering "if I changed asset X, what would the impact be on asset Y?" as well as a basis for discovering the root cause of unexpected results. An asset-based orchestrator addresses gaps in the Modern Data Stack by combining polyglot compute with a declarative, asset-based approach, enabling a unified control plane without requiring practitioners to revert to tasks and imperative programming. The software-defined asset is a natural unit for data management, lineage, and observability, declaring the order you want to create instead of describing the chaos that exists.