August 2023 Summaries

Migrating from dbt Cloud to using the free and open-source version of dbt-core and orchestrating it with Dagster is a viable alternative. Users can run their data pipelines in production on Dagster Cloud, which offers a curated onboarding experience for new users. The tool provides features such as automated data documentation sites, IDEs, scheduling, alerts, and unified stack integration. With Dagster, users can also create branch deployments for pull requests, test against production-scale data, and update their dbt docs on every deployment. Additionally, Dagster offers integrations with other modern data stack tools, making it a suitable solution for organizations looking to run the data transformation steps as part of a more fully-featured orchestrator.

Dagster Labs, a rapidly growing company, has been prioritizing effective hiring and scaling its team since the successful Series B funding in May 2023. CEO Pete Hunt joined James Mackey on The Breakthrough Hiring Show to discuss how businesses can leverage data, AI, and people-centric hiring practices to drive growth. During the conversation, Pete shared insights into the technological and business aspects of data management, the transformative potential of AI, and the importance of prioritizing team growth and well-being.

A Dagster tutorial on creating an ML pipeline for fine-tuning Large Language Models (LLMs) using LoRA and parameter-efficient techniques. The authors share their findings and demonstrate best practices in creating a clean production ML pipeline, including operationalizing and keeping the model up to date, monitoring quality, and automating the pipeline with Dagster's resources and asset-based coding. The tutorial covers topics such as choosing the right LLM, using notebooks to build fine-tuning models, converting notebooks to Dagster code, thinking in assets and resources, tokenizing data, building an ML pipeline, evaluating model performance, and automating the pipeline. The authors provide a comprehensive guide on how to create a production-ready ML pipeline with Dagster, making it easier for machine learning teams to streamline their workflows and improve productivity.

Podcast: Adventures in DevOps - Testing and Development in the Data Domain` Podcast hosts Jonathan, Jillian and Will chat with Pete Hunt, CEO of Dagster Labs, as they explore the challenges faced by engineering teams in the evolving landscape of cloud technology and open source. They discuss misalignment between individual engineers and company goals, managing complex systems like Airflow, advancements in data pipeline scheduling, and the need for a higher-level approach. The importance of open source is also highlighted along with dynamics of working with venture capitalists.

Podcast: Engineering Hard Choices` is a podcast episode featuring Pete Hunt, CEO of Dagster Labs, where he shares insights on building and leading a data engineering team and making hard choices such as selecting an app's tech stack. He discusses topics like job search strategies, self-learning, and the relevance of advanced degrees in the field of software development. Additionally, he touches upon his experiences with building Instagram using client rendering and starting a startup from scratch before selling it to Twitter. The episode also explores the concept of strategic happiness, professional development for engineers, and the impact of data scientists on the creation of next-generation data orchestration tools.

The company behind the open-source project Dagster and its commercial product Dagster+ is changing its name to Dagster Labs, effective immediately. The change aims to communicate the company's focus on building a company around its mission, rather than just its sole product, Dagster. The company's team and leadership are Nick Schrock and Pete Hunt, who will continue to lead the company under the new name. The rebranding also includes some minor changes in branding and domain names. Despite the change, nothing else is expected to fundamentally alter the company's operations or goals, and its community and customers can expect continued support. The announcement comes as part of a broader effort to clarify the company's identity and mission, with the goal of serving its open-source community and cloud customers more effectively.

A Dagster project is presented, which builds an outbound reporting pipeline for sending email reports with custom user-specific charts. The solution uses data engineering patterns to orchestrate the email delivery, making it straightforward to build and maintain. The pipeline consists of several assets, including `monthly_reservations`, `property_analytics`, and `historical_bar_charts`, each with its own logic and dependencies. The project also includes a job that wraps these assets and an op to send emails using a configurable resource called `EmailService`. To deploy the pipeline in production, considerations are made for data access, I/O Manager, fanning out or multi-partitioning, error handling, logging, and testing. The project showcases how Dagster can be used to build scalable and maintainable data pipelines.

Dask is a flexible library for parallel computing in Python that can be used to speed up data engineering and machine learning tasks. Dagster, a platform for building data pipelines, can be used with Dask to automate computations and make pipelines faster. A Dask resource can be defined using Dagster's resources feature to set up a Dask cluster, allowing for centralized configuration of the cluster across multiple assets. This enables simplification of pipelines by reducing redundant code in assets. The Dask UI can be accessed to monitor executions from Dagster, which are only available while the executions are in progress. Using Dask with Dagster allows for speeding up processes and building machine learning pipelines that utilize Dask's parallel computing capabilities. A machine learning pipeline using Dask involves generating synthetic data, splitting it into training and test sets, searching for the best classification model, scoring the model, and loading the assets into a code location or Definitions. The use of Dagster with Dask bridges business intelligence and data orchestration, allowing for the orchestration of unstructured data pipelines.

Type hints in Python are a powerful tool to promote clear and reliable code by providing the expected type of variables, function parameters, or return values. They can enhance code readability, prevent type-related errors, improve IDE support, facilitate large-scale projects, but are not enforced at runtime. Python's type hints are optional and work best when combined with tools like `pyright` for static type checking. Type hints and docstrings together provide a clear understanding of what types of data your function expects and what it will return. Python has many built-in modules that offer additional data types, and you can also define custom data types using classes to further improve code maintainability and reliability.

Environment variables in Python provide a way to configure applications in a non-hardcoded manner, enabling modification of application behavior without changing the actual code. They are especially important when parameterizing data pipelines in a production environment, allowing sensitive information like database credentials or API keys to be stored outside the codebase, enhancing security and making the code more portable and easier to manage. Python's built-in `os` module provides an interface to interact with the underlying operating system, including a dictionary-like object called `os.environ`, which allows access, modification, or creation of environment variables in a Python program. Environment variables are accessible within the same process but not between processes, so they require setting outside of Python for persistence across different sessions or processes. They can be used for configuration data that varies between deployment environments and sensitive data that should not be stored directly in the code. Using environment variables fosters consistency by keeping development and production environments consistent and helps prevent bugs by ensuring that application settings remain consistent across all environments.

Pete Hunt, CEO of Dagster Labs, discusses the importance of optimizing data orchestration to be aware of the assets it is computing. This is critical for efficient compute in large-scale data teams as they deliver production data products to meet business SLAs while managing cost and resources. Pete draws parallels with his work on React.js, emphasizing the need for asset-aware re-rendering in modern front-end applications.

In episode 109 of "Drill to Detail," Mark Rittman, founder of Rittman Analytics, converses with Nick Schrock, founder of Dagster Labs, about Dagster's significance in the modern data stack ecosystem and the concept of software-defined assets. The discussion highlights Dagster's innovative approach to data management and orchestration, emphasizing its role in maintaining data through a new, declarative methodology.

Pete Hunt, CEO of Dagster Labs, was interviewed on the Scale Up Show podcast by Ryan Staley. Pete discussed his work at Facebook and Twitter, including meetings with Mark Zuckerberg and creating a high valuation to sell his previous company. He also talked about building a company to revolutionize data harnessing and shared his thoughts on Threads as a product. Additionally, he provided advice for achieving perfect product-market fit. The interview is available in the Dagster community Slack or can be discussed on Github.

Dagster has become the industry standard for orchestration of SQL transformations within a warehouse, particularly with its integration with dbt. The latest release (1.4) focuses on making Dagster's dbt integration more flexible and easier to get started. Unlike other general-purpose orchestrators like Airflow, Dagster's core design principles align well with dbt's approach to data pipelines, allowing for faithful orchestration of dbt models. Additionally, Dagster compensates for dbt's limitations by connecting the models in a dbt project to other kinds of data assets, such as tables ingested using tools like Fivetran or machine learning models. Dagster provides a full set of orchestration features, including flexible scheduling options, observability, partitioning, and alerting. The tool also allows for self-deployment on-premises, unlike dbt Cloud, which is a proprietary cloud service with no open-source equivalent. By using Dagster, teams can orchestrate their data pipelines more efficiently, ensuring that their data assets are materialized to meet their SLAs.