Company
Date Published
Author
Sandy Ryza
Word count
1627
Language
English
Hacker News points
None

Summary

Data teams often spend a significant amount of time fixing broken assets and pipelines, with data engineers dedicating up to 80% of their day to debugging failures. This issue is particularly prevalent in large organizations where the root cause of an issue can lie in a source table managed by a different team. To tackle this problem, Dagster Labs has developed an orchestration platform designed to directly address productivity issues faced by data and ML engineers. The platform's critical role in debugging lies in its orchestrators, which schedule and order computations, making it easier to untangle what has happened and understand what was supposed to happen. Dagster's asset layer helps teams think about assets first and tasks when needed, providing a blueprint for analysis that guides human operators to debug faster. The platform's UI is designed to dramatically accelerate debugging by rapidly finding the root cause of failures and providing easy access to logs and metadata. By empowering organizations to build productive data platforms, Dagster aims to reduce the time spent troubleshooting issues and increase velocity.