August 2020 Summaries

Nick Schrock, the founder of Dagster Labs, shared his expertise on how to sell new tech concepts to developers in a recent podcast episode. He discussed his blueprint for engineering excellence and provided insights into the future of open-source projects like Dagster. Sandy Ryza, another guest, talked about the impact of data scientists on the creation of next-generation data orchestration tools. The podcasts cover various topics, including data science, machine learning, and open-source development.

Dagster is a new type of workflow engine, called a data orchestrator, designed to manage complex data systems. It introduces a data-aware, typed, self-describing logical orchestration graph that models the implicit structure in every data application and platform. This graph enables structured programming, testability, reliability, and observability, making it easier for data teams to collaborate, debug, and deploy their systems. The orchestrator targets full-stack engineers, data scientists, and platform teams, providing flexibility and adaptability to run any tool, use any storage, and deploy to any infrastructure. It addresses the pain points of traditional workflow engines by introducing a more structured programming model that exposes a richer, semantically aware graph. By adopting Dagster's API, users can build reusable components, manage complexity, and achieve end-to-end ownership of data production. The system is designed to work with heterogeneous tools, teams, and infrastructure, providing process isolation, deployability, and operational efficiency. It aims to empower data practitioners to become participants in a software engineering process, enabling them to contribute to the development of their own applications and tools.