June 2026 Summaries

Dagster, a data orchestration tool, has been creatively utilized by the community for various projects, including public dataset exploration, infrastructure monitoring, and workflow automation. Highlighted projects demonstrate the versatility of Dagster, such as Edwin Weber's integration of Dagster with dbt and DuckDB for a budget-friendly data engineering setup focused on Danish parliamentary data, and Parag Ekbote's development of a Python library to integrate Dagster with Hugging Face Datasets for streamlined dataset workflows. Both users emphasize Dagster's asset-based architecture, strong community support, and extensive documentation as key factors in its effectiveness for data orchestration tasks. Challenges faced, such as managing streaming data and metadata, were overcome with Dagster's flexible design, making it a robust choice for complex data projects.

The comprehensive guide explores the intricacies of operating data orchestration layers within open data platforms, emphasizing the significance of integrating DevOps and GitOps practices for efficient deployment and management. It discusses the challenges of scaling data engineering, especially in the context of AI and generative AI, where data use intensifies, highlighting the role of GitOps in automating infrastructure changes through code repositories. The guide delves into best practices for deploying data orchestration tools like Dagster, focusing on separating business logic from technical infrastructure and emphasizing the need for partitioned, incremental processing to optimize costs. It addresses the importance of governance and testing in ensuring data quality and reliability, advocating for the use of code locations to manage stateful and stateless processes effectively. Additionally, it highlights the need for a multi-tenancy approach to support various teams' autonomy while maintaining a unified data platform strategy, ultimately positioning orchestration as the operating system of a modern data platform that integrates diverse tools and functions seamlessly.

The Orchestration Maturity Model is introduced as a framework to help organizations transition from job-centric to asset-centric orchestration, emphasizing the limitations of traditional job-based systems in addressing critical data dependencies, freshness, and usability. The model outlines different levels of maturity, with Level 3 marking a shift toward treating data as a first-class object, enabling organizations to better manage data assets and automate processes based on data state. Dagster is highlighted as a tool that facilitates this transition by shifting the focus from tasks to data flow, making dependencies explicit and integrating quality checks into the orchestration process. This shift not only addresses common failure modes in traditional systems but also empowers teams to manage data workflows more effectively, reducing reliance on central data teams and enabling self-service capabilities. The transition to an asset-centric system requires architectural changes that align with organizational goals, providing a roadmap for improving data platform maturity and unlocking new operational and organizational capabilities.

Dagster, an open-source data orchestration platform, is celebrated for its versatility and ability to transform diverse data engineering projects into coherent systems. The community has creatively utilized Dagster for a range of applications, including public data exploration, infrastructure monitoring, research automation, and internal tooling. The platform's Pythonic nature, coupled with its capability to handle various logistics of data engineering such as orchestration, scheduling, and asset management, makes it particularly appealing. A featured project involves constructing a vertical data stack using open-source data primitives, highlighting how Dagster facilitates efficient pipeline creation and management. Key benefits include the ability to build pipelines across domains like budget and transportation using advanced data tools while embedding trust into the data lifecycle. The platform's composability allows users to design modular and scalable data workflows, encouraging experimentation and innovation. While initially challenging due to its flexible design, Dagster proves to be a powerful tool for building complex data systems once users become accustomed to its features.

Text-to-analytics (T2A) offers self-service access to data, with usability, governance, and trust being key to its adoption. Brooklyn Data, a data consulting firm, shares its experience with deploying Compass, a text-to-analytics tool integrated within Slack, which works by using a large language model to interpret user queries and retrieve relevant data. The firm emphasizes the advantage of Compass in providing a familiar, interactive platform for querying data without requiring a separate login or interface, while maintaining centralized governance. By integrating with Snowflake, Compass offers improved searchability and operational efficiency for Brooklyn Data, reducing the time needed to retrieve information from their professional services automation software. Users can provide direct input to the model, enhancing its contextual understanding and performance. Compass's seamless integration into Slack allows team members to make data-driven decisions quickly, and its success at Brooklyn Data has led to plans for broader adoption within the company and a future expansion to marketing data.