October 2020 Summaries

Dagster and dbt are complementary technologies used to build data transformations in an intuitive, testable, and versioned environment. dbt enables users to define well-defined data transformations using templated SQL, forming a dependency graph between the transformations. Dagster orchestrates these transformations alongside other technologies, providing operational and data observability capabilities. The integration allows engineers to monitor dbt model execution in the same tool they use to monitor other technologies, with historical and longitudinal views of operations. This enables explicit modeling of dependencies between dbt models and processes that use other technologies, scheduling and executing pipelines that include both dbt and other technologies, and monitoring dbt models alongside other processes. The Dagster-dbt integration is available in the dbt docs and dagster-dbt integration documentation.

At Good Eggs, a local organic grocery store, data modeling is crucial to manage marketing, purchasing, customer buying patterns, personalization, retention, and logistics strategies. The team uses a diverse stack of tools, including dbt, Snowflake, Mode Analytics, Jupyter Notebooks, Stitch, and custom Singer taps. However, managing this complex system poses challenges, such as tracking dependencies, ensuring data quality and freshness, and controlling resource usage. To address these issues, Good Eggs adopted Dagster, an orchestrator framework that provides features like support for custom data types, structured metadata, and testability, ultimately transforming their data platform team's software engineering lifecycle.

This case study from Good Eggs highlights the importance of data quality and reliability in a data-driven organization. The company uses Dagster, a data orchestration platform, to manage its data ingest process. By implementing custom data frame types, they are able to run automated type checks on their data, which guarantees invariants such as non-null values and categorical data types. This approach ensures that bad data is caught quickly, reducing the risk of polluting downstream data artifacts. The use of structured metadata also provides benefits for human analysts, enabling them to diagnose data quality problems more efficiently. Overall, Dagster's support for custom data types and structured metadata has improved the reliability and efficiency of Good Eggs' data ingest process, reducing latency from days to under an hour.