June 2025 Summaries

Dagster 1.11 "Build Me Up Buttercup" introduces significant improvements to pipeline-building with new features like Components and the dg CLI, designed to enhance user experience and orchestration capabilities. Components, now stable, offer configurable and reusable building blocks in YAML or Python, allowing for effortless definition of Dagster assets and pipelines with minimal boilerplate. The dg CLI acts as an all-in-one command-line tool that supports code scaffolding, local development, and job execution while offering introspection and validation of definitions. The update also includes a streamlined create-dagster command for quick project setup, improved orchestration features such as partial retries and enhanced backfill policies, and expanded ecosystem integrations, including Fivetran, dbt Cloud, and Airflow. The release emphasizes collaboration with the community and design partners, offering acknowledgments to contributors and encouraging feedback through platforms like Slack and GitHub.

Dagster's introduction of the `dg` tool and Components marks a significant advancement in data tooling, allowing for the creation of well-organized projects and the development of domain-specific languages (DSLs) through YAML-based configurations. DSLs, like SQL, offer a focused, high-level abstraction tailored to specific domains, which makes them powerful despite their limited scope. Dagster Components enable users to abstract complex processes, such as data extraction and transformation, into reusable, declarative building blocks, simplifying workflow implementation without delving into low-level technical details. This approach contrasts with traditional imperative data pipeline tools by modeling systems as collections of assets rather than sequences of steps. Components allow for a customizable user interface that balances abstraction and flexibility, empowering data teams to work more efficiently while maintaining clarity and consistency. Although DSLs cannot encapsulate all complexities of data systems, Dagster's design allows for the integration of DSLs with imperative Python code, providing a scalable platform that supports both abstraction and detailed implementation.

Dagster offers flexible code location management, allowing developers to organize their project in various ways. A single code location can work well for early-stage projects or smaller teams with a more general approach, while team-based locations are suitable for larger teams with different development cadences. Tool-based locations group code by function and dependencies, promoting clarity and clean boundaries between concerns like ingestion, transformation, and reporting. Dedicated code locations can be used to isolate critical pipelines or assets, allowing for fine-tuning, isolation, or control over specific sets of assets. Decoupling shared code, such as custom Dagster resources, utilities, or helpers, from specific asset definitions can also improve organization and management. Ultimately, there are no hard rules, and the best approach depends on the team's structure, workflows, and how the organization manages code.

Connect 211, a non-profit organization providing health and human services data, built a platform to standardize and publish data from 211 call centers across the US. They needed a data orchestration platform that could efficiently connect people with information they need. The company initially used Airflow but faced issues with visibility, diagnostic difficulties, complexity at scale, and integration challenges. After evaluating other options, they chose Dagster due to its enterprise-grade scalability, big-feature richness, and tight integration with dbt. With Dagster, Connect 211 built a robust data orchestration system that connects health and human services data sources, standardizes data, and publishes it to a user-friendly directory. The platform has improved pipeline reliability, observability, and dramatically reduced troubleshooting time. It provides standardized data, scalable architecture, and opportunities for advanced applications such as AI-powered entity resolution and sentiment analysis.

Big Cartel, an e-commerce platform designed for artists and small businesses, was struggling with a fragmented data system that lacked visibility, usability, and consistency. The company hired Patrick Trainer, a data engineering team of one, to transform its disconnected data silos into a cohesive, observable platform using Dagster at its core. With Dagster, Big Cartel now has a future-proof data platform that scales from internal analytics to customer-facing insights, providing a unified control plane and modern developer experience. The transformation has reduced cognitive load through automation, visualization, and proactive monitoring, making it easier for non-technical stakeholders to understand data lineage and dependencies. By implementing Dagster, Big Cartel has created an accessible, reliable, simplified data foundation that supports its growth and future capabilities.