July 2023 Summaries

The authors of the documentation for dbt (data build tool) have invested heavily in integrating it with Dagster, a data engineering platform. They found that a significant portion of users used dbt and were struggling with its performance on large-scale projects. The team identified issues with dbt docs, such as slow page load times and high memory usage. To improve performance, they implemented React Server Components and Next.js 13 in their project. This resulted in a significant reduction in page load time, from over 4.5 seconds to under half a second, and improved overall user experience. The implementation also reduced memory usage by approximately 20x. However, there are still some challenges with the current solution, such as patching Next.js to fix a bug and limitations on serving static sites from the root of the domain.

Pete Hunt, the current CEO of Dagster Labs, was recently interviewed on the A Geek Leader podcast. He joined as head of engineering in early 2022 and became CEO in November of that year. Pete has a diverse background, having previously co-founded Facebook's React.js open-source project and leading Instagram's web and business analytics teams before being acquired by Twitter. The podcast covered his experience working on React.js, building open source solutions, and the Data Orchestration domain. As he transitioned from an engineering leader to CEO, Pete emphasized the importance of being able to say "I don't know" while still inspiring confidence and mastering delegation.

Dagster version 1.4, codenamed "Material Girl", has been released on Dagster Cloud and in Open Source, introducing several key enhancements including improved dbt integration, evolving asset materialization, and a new UI layout. The release includes brand new capabilities for dbt users such as a new `@dbt-assets` decorator and scaffolding command line interface, allowing more control over how Dagster runs dbt projects. Additionally, the release introduces features like auto-materialization history, op-level concurrency limits, and experimental new features to improve ergonomics, making it more logical and accessible for users. The 1.4 contributors are thanked, and feedback is encouraged through various channels including the Dagster community Slack and GitHub discussion.

Sandy Ryza, lead engineer on the Dagster project, presented at the DATA + AI Summit in San Francisco on June 8th, discussing "The Future of Data Orchestration: Asset-Based Orchestration." Sandy's thesis proposes a new pattern and way of thinking for data orchestration, introducing asset-based orchestration with data freshness sensors to trigger pipelines. The presentation covered topics such as data pipelines, asset updates, workflow engines, and asset observability, aiming to update the traditional patterns used since the 1980s. The talk was recorded and made available for viewing, providing an opportunity for attendees to learn about this new approach to data orchestration.

Training Large Language Models (LLMs) requires contextual data, which is often dispersed across various sources. To maintain this data's freshness, a robust pipeline is necessary, transcending ad hoc scripting. Dagster can orchestrate services involved in LLM training by running ingestion tasks, transforming and structuring data, and making it available for the LLM. When paired with an ingestion tool like Airbyte and a framework for language models like LangChain, the task of making data accessible to LLMs becomes feasible, maintainable, and scalable. The pipeline involves three steps: data ingestion using Airbyte, configuring the pipeline in Dagster, and loading the data. The final code can be found on Github, and prerequisites include Python 3, Docker, an OpenAI API key, and specific dependencies. The example showcases how to use Airbyte and Dagster to bring data into a format that can be used by LangChain for question-answering applications. The pipeline can be materialized from the command line or deployed in production using Dagster's features.