December 2023 Summaries

Data engineering has transformed and adapted to the way we use data in everyday life so much over the past decade, according to Nick Schrock, founder of Dagster Labs. Data engineering is no longer just a buzzword, but a vital part of how we handle the vast amounts of data created every day, which now totals 328.77 million terabytes. The field has evolved to meet the changing needs of data-driven applications and organizations, with experts like Nick Schrock discussing the current state of data engineering and its impact on various industries. As the amount of data continues to grow exponentially, data engineers are facing new challenges in terms of data orchestration, which is becoming increasingly important for businesses looking to extract value from their data assets.

The Dagster Open Platform (DOP) is an open-source, public repository for others to reference and use as inspiration. It applies best practices in data engineering and Dagster to build complex asset-based pipelines. DOP contains parts of the Dagster Labs business and is used by the team to practice what they preach. The project provides a valuable resource for Dagster users, growing data teams, and SaaS startups looking to build a reliable data platform. The Dagster Open Platform repository includes code snippets that demonstrate technical best practices and patterns implemented by the Dagster Labs team. The team has open-sourced their telemetry data transformation and analysis models using dbt, allowing others to learn from their implementation. DOP also provides environment helpers that leverage variables set by Dagster Cloud to determine which database or S3 bucket to read and write from. This helps with managing pipelines across different environments such as local development, branch deployments, and production. The team has defined a set of dbt-related objects, including a translator that maps dbt sources to upstream models and enriches metadata for each model. DOP is still in its early stages, but the team will continue sharing their pipelines and end-to-end solutions with the open-source data community.