April 2024 Summaries

The text discusses the challenges of working with multiple similar data assets in a data engineering context. It highlights the importance of maintaining code maintainability and avoiding duplication (DRY principle). The authors introduce the concept of Asset Factories, a design pattern that enables generating multiple similar asset definitions dynamically. They provide an example using Dagster, a platform for building data pipelines, which showcases how to create a customized `@asset` decorator using `functools.wraps`. This allows for dynamic definition of dependencies between assets and empowering data engineers to craft flexible, maintainable data pipelines tailored to their specific needs. The solution also extends the capabilities of the `@asset` decorator by injecting additional asset dependencies or resources, making it a powerful tool for creating reusable and flexible data pipelines.

Dagster+ is a tool that helps ensure reliable data by monitoring and detecting issues in data pipelines. It enables teams to track the freshness of their data, identify quality issues, and catch changes in schema. With Dagster+, teams can monitor both source data and output data from their pipelines, allowing them to detect potential problems early on. The tool offers a full set of data reliability features, including asset checks that can be used as a basis for alerting and reporting. Additionally, Dagster+ provides an asset health dashboard that integrates pipeline failures, data quality, and freshness into a single view, making it easier to standardize data reliability across the entire data platform.

Dagster+, a data engineering platform, introduces Branch Deployments to improve the development process for data pipelines. These deployments allow users to test and review changes in a staging or test environment before deploying them to production. With Branch Deployments, users can see which assets have changed, making it easier to identify potential problems and understand code changes. The feature also provides Change Tracking, which highlights affected assets and improves the testing and reviewing process. This enhances the development process by reducing friction and increasing quality, enabling teams to move faster while maintaining high-quality data pipelines.

Dagster+ Insights is a single pane of glass for understanding the state and cost of data platforms, providing both data platform owners and individual practitioners with visibility into the quality, timeliness, and cost of their entire data platform. It empowers users to see the impact of their pipelines on the whole platform and understand the overall health and spend of their data platform. The tool is designed to address the complexity of large, complex, and mission-critical data platforms that integrate dozens of systems and serve hundreds of stakeholders. With Dagster+ Insights, users can track trends for freshness, quality, and cost of their data platform, enabling proactive management across data platforms. The tool also enables users to visualize bytes billable in both data and dollar terms, filter metrics by tags and owner, and receive alerts when costs exceed a certain threshold.

The Dagster+ Catalog is a new built-in asset library designed to help data practitioners find and understand specific data assets in large-scale data platforms. Unlike traditional standalone data catalogs, which often fail due to the overhead of running metadata management systems, the Dagster+ Catalog integrates directly into the fabric of the Dagster platform, easing data discovery and documentation issues without adding complexity. The new catalog provides a context-rich view of data assets, allows for data exploration at the column level, and features a flexible organizational system, search UI enhancements, and integration with external assets, ultimately transforming data management with Dagster+.

This article explores the cost-effective and efficient solution of using Dagster and SkyPilot to orchestrate ML training jobs within a single data platform. The combination abstracts the resource acquisition and job execution through an intuitive declarative DSL, allowing data engineering to invite ML teams to bring their existing ML training and inference pipelines into Dagster and orchestrate them with minimal code changes. SkyPilot enables resilient and cost-effective AI/ML training jobs across cloud environments and regions by implementing the Sky Computing paradigm, where workloads can be transparently executed on one or more clouds, abstracting the provision of resources and execution of arbitrary workloads across cloud vendors while automatically maximizing cost savings and availability for users. The solution is particularly useful for organizations that want to execute AI training jobs in a cost-effective fashion with plug-in support for spot instances and automatic recovery from preemption.

The primary concern of data engineering is building and maintaining data assets, which are tangible entities that have meaning. Most tools in the data engineering tech stack take an asset-oriented approach, but orchestration tools often struggle to integrate with these assets, creating an impedance mismatch. This mismatch can lead to headaches such as clunky integrations, fragmented dataflows, and limited observability. To address this issue, Dagster was built as an asset-oriented data orchestrator, enabling super developer experience, decentralized data platforms, and a single pane of glass over the whole platform. Dagster+ builds upon these fundamentals by introducing new capabilities that leverage the lineage and metadata of assets, including data reliability, operational insights, data cataloging, and more. These capabilities aim to streamline data pipeline development and operation, reduce complexity, enhance transparency, and foster collaboration.

The Dagster team has released version 1.7 of their platform, which includes several enhancements to improve asset cataloging and organization. The new release introduces a "Love Plus One" theme, with features such as a new Asset Details page that centralizes important information about assets, improved tags for organizing assets into logical groups, and enhanced Asset Checks that allow for more flexibility in defining checks for data freshness and column schema changes. Additionally, the Dagster dbt integration has received updates to automatically collect metadata and pull in tests and resource tags as asset checks. The release also includes quality-of-life improvements to the UI and documentation, as well as a new module on dbt in Dagster University. The team thanks all community contributors since 1.6.0 for their contributions to this release.

Dagster has officially integrated with dltHub, an open-source Python library for building pipelines to ingest organic data sources into well-structured datasets. This integration expands Dagster's functionality by filling the gap in ingesting data from endpoints, APIs, and disparate systems. The dlt library allows developers to define their own sources in Python, making it a flexible tool for various use cases. The integration is built on top of multi-assets, allowing users to leverage existing dlt pipelines with Dagster's robust framework for building and orchestrating fault-tolerant jobs. This partnership aims to provide an intuitive developer experience for data ingestion pipelines from a variety of sources, combining the strengths of both technologies.

The author of this text, Nick Roach, and his team at Dagster Labs successfully migrated their data ingestion workflow from a managed ETL provider to Embedded ELT using Sling. The migration resulted in significant cost savings ($40,000 per year) and improved control over the ingestion process. The team created a testing environment, built replication configurations for both small and large tables, tested connections with the Sling CLI, and set up Dagster Embedded ELT. They also performed parity testing, swapped downstream tables to source their Sling data, and removed references to their old ETL provider from their Dagster project. The team found that Embedded ELT provided a lightweight toolkit for ingesting and loading data with Dagster, allowing them to define how data is synchronized from source to target data system. The migration was completed within two weeks, demonstrating the speed and efficiency of using Embedded ELT and Sling.