December 2022 Summaries

The text outlines the transition from using traditional ETL scripts to a more efficient and maintainable data pipeline using Dagster, an orchestrator platform that enables the use of Software-Defined Assets. It begins with a basic explanation of the ETL process—Extract, Transform, Load—and the challenges posed by conventional ETL scripts, such as issues with development velocity, scheduling, robustness, and testability. Using an example that fetches and visualizes Hacker News stories, the text demonstrates how to migrate these scripts to Dagster by wrapping scripts in a single software-defined asset, introducing retry and freshness policies, and creating dependencies between assets to increase development speed. Additionally, it discusses abstracting storage logic away from business logic using Dagster’s IO Manager to improve modularity and testability. The text encourages further exploration of Dagster's capabilities through its documentation and community support.

Declarative Scheduling is an exciting feature in Dagster that allows data engineers and analytics engineers to orchestrate dbt models with improved ground-breaking improvements. Unlike other orchestrators, Dagster uses software-defined assets (SDAs) which enables users to build a fine-grained understanding of the dbt graph, its upstreams, and its downstreams. This allows Dagster to provide scheduling guarantees that other orchestrators cannot. With Declarative Scheduling, users can specify freshness policies for their models, which simplifies the scheduling process and saves resources by reducing redundant model runs. Users don't need to explicitly add their models to a job or think about cron schedules, and Dagster will automatically ensure that all dependencies are met. The feature has been introduced in the latest Dagster release and is available as part of the dbt integration.

Sandy Ryza is releasing version Dagster 1.1: Thank U, Next, a significant update that includes support for declarative, asset-based scheduling, improvements to Fivetran, Airbyte, and dbt Cloud integrations, and more. Declarative scheduling models each data asset as a function of its predecessors and schedules work based on how up-to-date you want your data to be. This feature will make working with assets much more efficient by declaring at the asset level how frequently it should be updated. The new release also includes a sensor that can monitor the state of multiple assets, partitioning of Software-defined Assets, easier exploratory data analysis on notebooks, and environment variables for local development. Additionally, there are enhancements to Dagit (Dagster UI), including an Overview page with a live run timeline, a Deployment page for managing Dagster instances, and improvements to the Asset Graph and Asset Details page. The release also includes updates to integrations with dbt Cloud, Airbyte, Fivetran, and DuckDB.

Dagster 1.1 introduces a declarative, asset-based scheduling system that revolutionizes data pipeline management by focusing on the timely updating of data assets rather than traditional task workflows. This approach enables users to specify how up-to-date each data asset should be, and Dagster automatically schedules updates based on data freshness and change detection, thus reducing unnecessary computations. In contrast to imperative workflow-based systems like Airflow, which can be cumbersome and inefficient, Dagster's system models data assets as functions of their predecessors, allowing for more flexible and efficient scheduling. The platform offers features such as asset freshness policies, granular versioning, and partitions, enabling precise control over data updates and ensuring that data products remain current. Additionally, Dagster supports traditional workflow-based scheduling for those who prefer it, while providing tools for automatic asset materialization and integration with business rules, making it a versatile solution for modern data orchestration challenges.

Sean Lopp built a data pipeline and dashboard to compare resort conditions in Colorado, using Dagster to manage the process. He found Dagster's dataset-awareness and strong opinions on code structure made it ideal for his project. The tool allowed him to separate production, staging, and local development, making it easy to deploy and test his code. Dagster's architecture also provided better performance and scalability compared to other schedulers like Airflow. Lopp was impressed with Dagster's capabilities and is now helping teams evaluate the tool to see its benefits for themselves.

Sandy Ryza, a lead engineer on the Dagster project, joins Tobias Macey on The Machine Learning Podcast to discuss how his background in machine learning has informed his work on the Dagster project and its foundational principles that allow for collaboration across data engineering and machine learning concerns. Building complex machine learning models requires an orchestration engine like Dagster that understands the data it is manipulating, enabling the creation of repeatable workflows and managing experiment iterations. Ryza shares insights into how he balances flexibility with repeatability in ML projects and highlights innovative ways Dagster has been applied to machine learning. He also discusses lessons learned from working on orchestrating ML workflows and what's planned for future improvements in ML support within Dagster.