November 2023 Summaries

Dagster, an orchestrator for data assets, is witnessing rapid adoption across industries and use cases. Currently, users build complex pipelines with thousands of assets, which poses challenges on the front-end rendering enormous graphs. To address this, Dagster's UI team has implemented several optimizations to improve performance. These include virtualizing asset nodes and edges to reduce rendering lag, batching data loading into chunks, and caching layout computations using IndexedDB. Additionally, they have switched from a complex layout algorithm to a simpler one that can be more efficiently rendered, resulting in a significant reduction of rendering time from minutes to under five seconds for large asset graphs with 2,000 assets. The team is also exploring new UX features, such as improved navigation and group views, to further enhance the user experience.

The SimpliSafe team, a large organization with numerous analysts and business users, transformed its analytics workflows by deploying Dagster's composable framework and automating deployments. This redesign enabled the engineering team to focus on expanding the platform and building new features while maintaining minimal support for analysts. By abstracting away pipeline creation steps in a YAML Domain Specific Language (DSL), analysts can define sources, schedules, and transformations quickly and easily, with the option to include Python code when necessary. The resulting data platform provides tremendous leverage, allowing the team to focus on proactive efforts rather than reactive troubleshooting. Despite some limitations, the YAML DSL approach has enabled SimpliSafe to build a reliable and efficient data process that supports its growing analytics needs.

High-performance Python code is essential for data engineering tasks, as it can significantly impact the efficiency of processing large datasets. Data engineers must consider various factors such as storage and performance trade-offs, choosing the right data types, leveraging specialized structures like NumPy arrays, and optimizing code using techniques like vectorized operations, lazy evaluation, and generator expressions. By applying these strategies, developers can create high-performance Python pipelines that efficiently process data in-memory or through compute engines like Apache Spark or databases. Effective optimization of Python code is crucial for achieving better performance, reducing costs, and improving overall efficiency in data engineering tasks.

Today, Laura and Kevin are joined by Pete Hunt, who shares his insights on what it means to be an engineer, how to identify a good engineer, and his journey from engineer to CEO. `Pete has experience working at Instagram and Twitter, where he led the web team and helped build business analytics products. He also co-founded Smyte, an anti-abuse provider acquired by Twitter, before taking over as CEO of Dagster Labs in 2022. `Pete's expertise covers next-generation open source orchestration platforms for data assets, and he offers his unique perspective on the Zuck vs Musk debate. `The conversation is available on the That Tech Pod podcast, where listeners can also provide feedback and engage with the community through Slack and GitHub discussions.

The text discusses the use of Dagster and dlt to orchestrate data pipelines. dlt is an open-source Python library that allows declarative loading of messy data sources into well-structured tables or datasets through automatic schema inference and evolution. It simplifies building data pipelines by providing functionality to support the entire extract and load process, including scalability, robustness on extraction, and state management for incremental extraction. To start with dlt, users can install it using pip and then import it in their Python script to build a data pipeline. The project code is available on GitHub. In this example, we will ingest GitHub issue data from a repository and store the data in BigQuery using Dagster and dlt. We will use dlt to create a simple data pipeline and then orchestrate it using Dagster. Finally, we will add more features to this pipeline by using dlt schema evolution and Dagster asset metadata. The project code is available on GitHub. To run the pipeline, users need to install Dagster and dlt, create a service account credential for BigQuery, and then execute the commands specified in the text. The text also provides instructions on how to orchestrate MongoDB verified sources using Dagster, including setting up a Dagster project, creating an asset factory, defining definitions, and running the web server. In this example, we utilized the Dagster @multi_asset feature to create a dlt_asset_factory that converts each collection under a database to a separate asset allowing us to create more robust data pipelines. Both dlt and dagster can be easily run on local machines by combining them, which enables building data pipelines at great speed and rigorously testing them before shipping to production.