Company
Date Published
Author
Gleb Mezhanskiy
Word count
582
Language
English
Hacker News points
None

Summary

dbt Labs, a leader in the Analytics Engineering movement, has partnered with Datafold to enhance data pipeline management by integrating agile software engineering principles like unified data transformation, version control, and automated testing. This collaboration focuses on delivering high-quality data by deeply understanding data connections, identifying issues before production, and verifying accuracy. The integration introduces features such as column-level lineage for dbt models, which maps dependencies to track data production, transformation, and consumption, as well as Data Diff for dbt, allowing analytics engineers to assess how model updates impact data and downstream dependencies directly in platforms like GitHub or GitLab. Additionally, shareable impact reports facilitate stakeholder review of data and metrics changes, promoting collaboration across teams like finance and marketing. Datafold's tools are designed to handle complex lineage graphs and offer insights into the effects of updates on entire dbt pipelines, all available through a one-click integration with dbt Cloud or via an SDK for teams using dbt Core.