Snowflake Runs Your Data: Dagster Runs Everything Else

Snowflake is enhancing its platform with features like Dynamic Tables and Cortex to handle data transformation and freshness internally, while Dagster complements these capabilities by offering orchestration, lineage, automation, and cost visibility across data platforms. Dagster allows users to define assets directly from SQL files using a simple YAML configuration, making it an efficient orchestration layer for Snowflake. This integration helps in managing asset definitions, lineage tracking, and materialization history, which is particularly beneficial for teams working primarily in SQL. Dynamic Tables in Snowflake pose orchestration challenges, but Dagster addresses this by creating virtual assets and using sensors to trigger downstream processes once new data is available. This pattern can be applied beyond Dynamic Tables to other managed objects like views and materialized views. Additionally, Dagster+ Insights provides visibility into the costs of running assets on Snowflake by attributing query expenses directly to the assets responsible, making cost management more transparent. The collaboration between Snowflake and Dagster is showcased at the Snowflake Summit, where Dagster demonstrates its features and discusses its integration in data platforms.