Company
Date Published
Author
Colton Padden
Word count
692
Language
English
Hacker News points
None

Summary

Dagster's integration with the dbt Fusion engine, currently in beta, brings significant enhancements to data pipeline workflows, offering substantial performance improvements by compiling and parsing projects up to 30 times faster than dbt Core, thereby reducing development and iteration times. This integration eliminates the need for the dbt-core Python package, instead using dbt Fusion's standalone Rust binary, and adapts to the presence or absence of dbt-core by employing CLI commands for full functionality. Key features of this integration include state-aware caching, a SQL-aware language server, and enhanced metadata management with tagging and column-level lineage, which together offer hyper-fast iteration, smarter SQL editing, and efficient execution with only changed models being rerun. Dagster's support for dbt Fusion also necessitated modifications such as removing the hard dependency on dbt-core, updating log parsing to accommodate Fusion's new event format, and ensuring consistent asset materializations and metadata extraction regardless of the engine used.