May 2026 Summaries

Snowflake is enhancing its platform with features like Dynamic Tables and Cortex to handle data transformation and freshness internally, while Dagster complements these capabilities by offering orchestration, lineage, automation, and cost visibility across data platforms. Dagster allows users to define assets directly from SQL files using a simple YAML configuration, making it an efficient orchestration layer for Snowflake. This integration helps in managing asset definitions, lineage tracking, and materialization history, which is particularly beneficial for teams working primarily in SQL. Dynamic Tables in Snowflake pose orchestration challenges, but Dagster addresses this by creating virtual assets and using sensors to trigger downstream processes once new data is available. This pattern can be applied beyond Dynamic Tables to other managed objects like views and materialized views. Additionally, Dagster+ Insights provides visibility into the costs of running assets on Snowflake by attributing query expenses directly to the assets responsible, making cost management more transparent. The collaboration between Snowflake and Dagster is showcased at the Snowflake Summit, where Dagster demonstrates its features and discusses its integration in data platforms.

Dagster's migration from Pyright to Astral's new Python type checker, ty, significantly enhanced their type-checking process, delivering both performance improvements and the unexpected benefit of identifying runtime bugs previously missed by Pyright. As Dagster's monorepo expanded, with more than 100 packages and continuous contributions, Pyright's type-checking duration became a bottleneck, taking about 15 minutes per CI run. In contrast, ty reduced this time to just 1-2 minutes, thanks to its Rust-native parser and analysis engine. Despite initial concerns over ty flagging approximately 4,500 diagnostics, the stricter type inference exposed legitimate bugs and unsafe assumptions, such as in the Pandera schema, that Pyright had overlooked. The migration, executed via parallel coding agents over 100 incremental pull requests, transformed the process into a scalable effort, despite ty being pre-1.0 with some diagnostic noise. Ultimately, the switch to ty not only streamlined CI processes but also improved code quality by revealing potential runtime issues, making it a promising tool for the Dagster team.

The text explores the evolution and impact of Dagster, an open-source data orchestrator, in simplifying and enhancing data engineering workflows. Initially conceived by Nick Schrock in 2018, Dagster focuses on transforming traditional ETL processes into a more developer-friendly, code-first approach, linking data processing with business processes. It introduces a paradigm shift from DAGs and task-based orchestration to asset-based orchestration, allowing for a declarative, data-aware approach that integrates seamlessly into complex data ecosystems. The platform supports multiple data environments and workflows, offering flexibility through its composable architecture and open standards. Dagster's control plane centralizes metadata and serves as a unified dashboard, promoting transparency and collaboration among data engineers, platform teams, and business stakeholders. With its emphasis on developer velocity and ease of use, Dagster stands out as a tool that not only facilitates the orchestration of complex data environments but also aligns with modern data engineering principles, making it a valuable asset for organizations navigating the complexities of enterprise data systems.