May 2023 Summaries

The Data Platform Podcast features various episodes with discussions on data orchestration tools and their psychological impact on companies. Pete Hunt, CEO of Dagster Labs, joined Jason and Iva for an episode where the conversation branched into a discussion about company psychology. Another episode discussed Sandy Ryza's thoughts on the impact of data scientists on creating next-generation data orchestration tools. Additionally, Nick Schrock shared his blueprint for engineering excellence on the Tech Talks Daily Podcast, and John Rouda interviewed him in a separate episode discussing open-source, ML, and Dagster's future.

The decade of data engineering has arrived, and it's about taming the chaos of managing production data assets. Data engineering is a discipline that unites roles such as ML engineers, analytics engineers, and data platform engineers, creating an interconnected graph of data assets. The current state of data engineering is characterized by siloed tools and point solutions, leading to organization-wide chaos. A unified control plane, like Dagster's asset graph, can bring order to this chaos, providing benefits such as productivity, context, and consolidation. This new layer empowers teams to develop and operate their data assets confidently and efficiently, making it a critical infrastructure challenge of the decade.

Elementl, a company behind the open-source data orchestrator Dagster, has raised $33 million in Series B funding led by Georgian, with participation from new investors 8VC, Human Capital, Hanover, and existing investors Sequoia, Index, Amplify, and Slow. The new capital will accelerate Elementl's development and adoption of Dagster, which is gaining traction across various industries due to its ability to simplify data orchestration and support modern engineering standards. Despite the challenging macroeconomic environment, the data category remains strong, with global businesses such as Doordash, Flexport, and Aritzia adopting Dagster for their data pipeline needs. Elementl's unique abstractions and asset-first approach are resonating with data practitioners, leading to rapid adoption and growth of the open-source community. The new funding will enable Elementl to scale its go-to-market activities and continue product evolution, further solidifying Dagster's position as a critical enabler for data teams looking to build robust and high-performing data pipelines.

Building better analytics pipelines is crucial for data-driven organizations as they face increasing complexity and challenges in managing their efforts. The lack of a proper framework and tooling for managing critical data assets makes it difficult to navigate organizational complexity, respond to stakeholder requests, enable self-service on data assets, and collaborate with other data practitioners across the enterprise. An orchestrator platform provides a development process, observability, and performance optimization tools such as scheduling and partitioning, which are essential for scaling data analytics efforts. Data engineers often struggle with managing complex systems, duplicated code, and poorly documented requirements, highlighting the need for an orchestration tool to bring order to the codebase, provide a unified control plane, and simplify the mental model in data analytics. An orchestrator like Dagster brings a unique declarative asset-centric approach that simplifies the mental model, enables selective materialization, and provides features such as schedules, integrations, metadata analysis, partitions, and backfills, allowing data engineers to build with confidence, deploy with ease, and put their data pipelines on autopilot.

Modern data pipelines are responsible for applying consistent computations to diverse batches of data. However, when dealing with a large number of data assets from different sources, understanding their lineage and keeping track of up-to-date information becomes challenging. To address this issue, Dagster has introduced dynamic partitioning, a strategy that enables a single pipeline to process items selectively from a data collection rather than managing separate parallel pipelines for each asset in the collection. This feature offers flexibility and declarative data management capabilities. By declaratively defining partitions, users can detect new files, reprocess corrupted data, run backfills, track execution progress, and simplify incremental updates. Dagster's dynamic partitioning allows users to model a data collection as a single, dynamically partitioned asset, providing granular control over the pipeline and high-level observability of the data lineage. This enables efficient processing of large datasets while maintaining a simplified and condensed view of the history of the data collection.

Dagster, an orchestrator for the data engineering development lifecycle, has recently improved its error surface in both Dagster Open Source and Dagster Cloud with enhanced features to provide more context and actionable steps for debugging and dealing with errors. The new capabilities allow Dagster users to receive detailed error messages including logs, context about the environment where the job was running, and suggested next steps to investigate further. This is particularly useful when errors occur in complex distributed tech stacks, such as Kubernetes pods running Dagster jobs, which may fail due to issues like missing secrets or incorrect system architectures. The enhanced error reporting provides a better developer experience by helping data engineers rapidly pinpoint issues and resolve problems more efficiently.

The text discusses the differences between Airflow and Dagster in terms of their configuration systems. Airflow uses a complex system with connections, variables, hooks, and operators, while Dagster uses a simpler resource abstraction system that makes integration dependencies explicit and handles UI rendering automatically. The example code shows how to rewrite an Airflow hook as a Dagster resource and operator, highlighting the differences in syntax and structure between the two systems. Additionally, the text touches on the templating system used by Airflow and its potential for confusion, whereas Dagster's system is more straightforward and user-friendly. Overall, the article aims to provide insight into the configuration systems of both platforms and how they compare in terms of developer ergonomics.

Software engineering teams often struggle to balance quality and speed in their product development, leading to a trail of user pain and a fragile codebase. To address this, companies must prioritize quality by adjusting processes and culture, focusing on delivering high-quality work and reducing the time spent fixing issues in production after each weekly release. This can be achieved through implementing dogfooding sessions, where engineers use their own product to identify bugs and improve it, as well as introducing automated testing and observability tools to track performance and detect potential issues before they impact customers. Additionally, companies must cultivate a culture of quality by educating the team on its impact, recognizing improvements publicly, and leading by example. By adopting these strategies, companies can improve their products and avoid spending excessive time on maintenance and bug-fixing, ultimately delivering better user experiences and driving business success.