July 2026 Summaries

Dagster, a data orchestration platform, has inspired a variety of innovative community-driven projects, highlighting its versatility and adaptability across different domains. Users have employed Dagster for tasks such as exploring public datasets, automating research workflows, and developing internal tools, reflecting the diverse creativity within the community. The platform's asset-first approach, which emphasizes data lineage and interconnectedness rather than just task scheduling, has resonated with data engineers like Kevin Longe and Boris Resnick, who have integrated it into projects like SQLBuild and complex machine learning pipelines. SQLBuild, developed by Longe, focuses on fast iteration and seamless integration with Dagster's asset model, while Resnick has utilized Dagster to orchestrate AI-driven drone fleet management and machine learning pipelines that require intricate data dependencies and resource management. Both practitioners praise Dagster's ability to enhance visibility, testing, and orchestration without hindering user experience, and they advise new users to start with simple setups while gradually scaling complexity as needed.

A tiered AI classification system was developed in nine days at Group 1001, a financial services company, to automate the classification of data across 62 databases with over a million columns as part of a large migration to a Snowflake account. The system combines deterministic regex rules and Snowflake's native classification tools to label columns effectively and affordably, addressing the challenges of maintaining consistency and accuracy in data governance. Dagster was used for orchestration, enabling a multi-stage pipeline that is legible and maintainable by a single person, even with context switching. The system features a human review workflow and ensures safe corrections through a polling loop, while the orchestration work was streamlined, allowing more focus on the core classification problem. The use of Dagster minimized the need for extensive orchestration code, making the rapid development and deployment possible.

Nick Schrock, the founder of Dagster, announced that Prefect is acquiring Dagster Labs, marking the convergence of two previously competing companies. Prefect has been favored by machine learning and platform teams for its dynamic workflow capabilities and user-friendly Python API, while Dagster has been popular among data teams for its advanced partitioning, lineage, cataloging, and scheduling features. The merger aims to leverage the strengths of both companies to better serve a wider range of orchestration needs. Dagster will remain central to the combined company's strategy, with active investment and development continuing under the Dagster brand. The transition will see about 40 Dagster team members joining Prefect, ensuring continuity in service and support for existing users. Schrock also announced his departure from the project, expressing pride in Dagster's development and optimism for its future under the new combined team.