Company
Date Published
Author
Pete Hunt
Word count
929
Language
English
Hacker News points
None

Summary

The primary concern of data engineering is building and maintaining data assets, which are tangible entities that have meaning. Most tools in the data engineering tech stack take an asset-oriented approach, but orchestration tools often struggle to integrate with these assets, creating an impedance mismatch. This mismatch can lead to headaches such as clunky integrations, fragmented dataflows, and limited observability. To address this issue, Dagster was built as an asset-oriented data orchestrator, enabling super developer experience, decentralized data platforms, and a single pane of glass over the whole platform. Dagster+ builds upon these fundamentals by introducing new capabilities that leverage the lineage and metadata of assets, including data reliability, operational insights, data cataloging, and more. These capabilities aim to streamline data pipeline development and operation, reduce complexity, enhance transparency, and foster collaboration.