June 2022 Summaries

Nick Schrock is the founder of Dagster Labs, a company that introduces software-defined assets, a new approach to managing, maintaining, and orchestrating data declaratively. Recently, Nick shared his expertise on various podcasts, including Value Driven Data Science - The Impact of Data Science on Data Orchestration, where he discussed the impact of data scientists on creating next-generation data orchestration tools. Additionally, Nick was interviewed by John Rouda on A Geek Leader podcast, where they talked about open-source, ML, and the future of Dagster. Nick's expertise has also been featured in Tech Talks Daily - Data, Decisions, and Dagster, where he shared his blueprint for engineering excellence.

I'm excited to join Elementl as the lead engineer, where I'll be working on Dagster, a project that's revolutionizing data engineering. The company was founded by Nick Schrock and has assembled an incredible team of talented engineers, including many former colleagues from my time at Facebook. What drew me to Dagster is its focus on fundamentals, programming model, and integrations with other systems. These parallels with React, another project I'm familiar with, made it clear that Dagster is moving data engineering in the right direction and has a path to widespread adoption. With Elementl's rapidly growing community, production-ready OSS project, and upcoming commercial offering, I couldn't be more excited about joining this team and contributing to the development of Dagster.

The modern data platform relies on dbt, a staple of the industry, but also requires tools to manage and monitor its various components. Python plays a crucial role in this ecosystem, particularly when it comes to executing complex computations or interacting with external APIs. Dagster, an orchestrator tool, provides a powerful solution for managing these workflows by offering software-defined assets, which enable cross-technology lineage, declarative data pipelines, detailed historical metadata, and the ability to orchestrate arbitrary Python within dbt projects. By leveraging these features, engineers can define and schedule jobs that update parts of their global graph, view the lineage of data as it moves between tools, and monitor the current state of all data assets in their platform. This approach simplifies the process of managing complex data workflows and provides a more efficient and scalable solution for companies looking to transform their data platforms.

We're excited to release version 0.15.0 of Dagster, which marks software-defined assets as fully stable and ready for primetime. Software-defined assets are a layer on top that links computations to long-lived objects, allowing users to build their Dagster jobs in a way that declares ahead of time what assets they produce and consume. This feature enables far-reaching lineage, improved code ergonomics, and the ability to operate assets directly. Additionally, we've made improvements to job retries, partition & backfill pages, left navigation pane in Dagit, Python API enhancements, and more. New features include asset groups, which organize software-defined assets into smaller groups that cut out irrelevant assets; configuration support for software-defined assets; and graph-backed assets, which enable the reuse of ops across different assets and jobs. We've also reworked our dbt integration to allow seamless interaction with tools like dbt, including rematerializing arbitrary sets of models from Dagit and scheduling jobs that run subsets of a dbt graph. Furthermore, we've introduced top-level inputs for Dagster jobs, generic outputs, run retries, metadata on jobs, a redesigned left navigation pane in Dagit, new backfills and partitions experience, and improvements to working with configuration and bulk actions on runs.