June 2023 Summaries

Dagster Cloud has introduced two new self-serve plans, Solo and Team, to replace its existing Standard plan. These plans offer event-based pricing (Dagster Credits) and per-seat pricing above certain limits. The Solo plan costs $10 per month for 5,000 Dagster Credits and one developer seat, while the Team plan offers 20,000 Dagster Credits and three developer seats for $100 per month. The new pricing model aims to provide a more predictable and affordable experience for users, with the ability to add more seats as needed. Existing customers can continue on their existing pricing model, but new signups will be subject to the new prices. The changes were made based on feedback from the community, which highlighted issues with the previous pricing model, such as charging customers while the service was idle and penalizing highly parallel use cases.

The text discusses the use of MotherDuck, a cloud service that hosts DuckDB tables, to replace an existing in-memory storage system. The author provides a step-by-step guide on how to connect to MotherDuck from the DuckDB CLI and integrate it with Dagster's IOManager abstraction. This allows for easy collaboration and scalability without touching business logic. The project demonstrates the benefits of using MotherDuck over S3 and Parquet, providing a robust, team-friendly data warehouse solution.

The Dagster Master Plan outlines the company's goals to accelerate software engineering best practices adoption among data teams globally. The project aims to improve data pipeline management, reduce complexity, and increase developer happiness by providing a unified asset graph that combines lineage, metadata, and operational history into a single system of record. To achieve this, Dagster plans to focus on three priorities: flattening the learning curve by streamlining its onboarding experience, evolving the orchestration category by integrating new types of metadata, and accelerating commercial success by delivering features above and beyond traditional orchestration capabilities. The ultimate goal is to make software engineering best practices widely adopted among data teams worldwide.

A backfill is a process in data engineering where historical parts of a data asset are updated or filled in using incremental updates, typically to maintain consistency and accuracy. Backfills are often necessary when changes are made to the underlying data source or code that generates the data, or when new data assets are added to a pipeline. The process can be complex and requires careful planning, execution, and monitoring to avoid issues such as resource overload, cost overload, and getting lost in the middle. Using partitions to organize data can make backfills easier by allowing for parallel processing and tracking of dependencies between data assets. A step-by-step guide for running a backfill includes managing data organization, planning, launching, monitoring, and verifying results.