April 2023 Summaries

Dagster version 1.3 is now available on Dagster Cloud and in Open Source, featuring several key improvements including the official induction of Pythonic Config and Resources as a permanent part of the system, enhancements to Software-Defined Assets for tracking changes, auto-materialization, partitions, and backfill functionality, and updated documentation. The new release includes UI improvements such as clearer labels for tracking changes, a new page for monitoring asset backfills, and reorganized Dagster Cloud docs. The team is grateful for community contributions to the project, who provide valuable input by suggesting new features, submitting PRs, and helping identify and document bugs.

The Public Utility Data Liberation (PUDL) Project, developed by Catalyst Cooperative, aims to make valuable public energy data readily accessible and user-friendly for those working to decarbonize the energy system. Initially using Python and pandas, the team faced challenges such as a burdensome process for adding new data sources, lack of parallelism, and difficult access to interim outputs from the ETL pipeline. To overcome these issues, they adopted Dagster, an open-source data engineering solution that enabled them to create a declarative approach, simplify their workflow, and accelerate development iteration cycles. With Dagster, Catalyst Cooperative can now add new data sources with relative ease, publish cleaned versions of tables, and make interim assets available to users, ultimately improving their ability to scale up the project and integrate more diverse datasets.

This guide is for beginners who want to start their first data engineering project with a basic understanding of Python. It focuses on using Dagster, an open-source solution for data orchestration, and provides a step-by-step approach to creating successful data pipelines. The guide covers setting up the project's root directory, launching a virtual environment, installing Dagster and scaffolding an initial project, declaring assets in Dagster, and understanding serialization in Dagster. It also introduces software-defined assets, which enable a declarative approach to data management, making it easier to manage and organize code. The guide provides examples of creating two assets: hackernews_top_story_ids and hackernews_top_stories, and demonstrates how to run the pipeline and create assets using Dagster's user interface.

The Empirico company, which specializes in drug discovery and development using human genetics and data-centric approaches, has implemented Dagster as part of their data infrastructure to support large-scale, multi-cloud computing. With a team consisting of data engineers, data scientists, bioinformaticians, and biologists, they aimed to improve the pace of drug discovery by speeding up the analysis of genetic data. Empirico built a bespoke interface to integrate with cloud compute layers, using standard big data tools like Spark, Pandas, Databricks, and NumPy. The team evaluated various orchestration solutions before adopting Dagster, which provided dynamic capabilities for branching and conditional execution. By leveraging Dagster's StepLauncher abstraction, they were able to transparently move op code to remote execution environments and trigger it from a Dagster step worker. This allowed them to manage compute environments dynamically, combining datasets and returning results based on user selections. The adoption of Dagster has transformed how Empirico can support big data analyses, enabling the team to introduce new compute providers without interrupting workflows and improving observability across the entire platform.

Meltano is an open-source tool that aims to solve the problem of ELt (Extract, Load, Transform) in the new world of SaaS. It provides a simple premise: writing any data extraction program to pull data from a source and writing any data loading program to push your data into a destination. Meltano was built on top of the Singer specification and has added an SDK for building new integrations, a configuration wrapper, and an integrations hub to support the community of Singer users. It offers over 550 integrations and is being prepped for its cloud service launch. The author of this blog post used Meltano to ingest data from GitHub and store it in a Dockerized Postgres database. They then added this pipeline to Dagster, an open-source orchestration platform, using the dagster-meltano library. This integration allows users to execute Meltano commands from within Dagster, enabling features like scheduling, dependency management, end-to-end testing, partitioning, and more. The author demonstrated three ways to run Meltano from Dagster: importing jobs from Meltano projects, issuing a meltano_run_op() command, and using the meltano_command_op() function to make other configuration changes programmatically. The guide aims to help anyone looking to tap into Meltano's capabilities as part of a Dagster managed project.

The Dagster project is releasing new features in its upcoming version 1.3, focusing on improving the config and resource systems to make it more Pythonic and user-friendly for modern users. The changes aim to remove the need to interact with the old config type system, provide strong type hinting and type checking, simplify existing APIs, and improve observability. Resources are now defined as classes with Pydantic attributes, making it easier to access resources in ops, assets, schedules, and sensors. The new resource system also allows for top-level binding of resources at the project level, eliminating the need to thread resources throughout a project. Additionally, Dagster is introducing changes to its UI to better represent resources as first-class components of the system, making it easier for developers to track down which assets rely on external services and drill down into their configuration. The new features are part of Dagster's efforts to become more ergonomic and user-friendly, with a focus on supporting modern Python users.