November 2020 Summaries

This is a case study about using Dagster, an open-source data orchestration platform, to improve data observability at Good Eggs. The company found that traditional debugging methods were too manual and unsustainable, so they integrated Dagster's asset catalog feature to track the history of their data assets. This allowed them to surface details about model runs in the Dagster UI, enabling analysts to view the lineage of their tables themselves without involving the platform team. The asset catalog also enabled tracking of trends and longitudinal charts for future wins, such as identifying developing bottlenecks before they become problems. By representing all meaningful side-effects of computations as assets, Good Eggs aims to achieve a completely horizontal view of their data assets, making it easier for analysts to track results and close the loop on the analytics workflow.