August 2022 Summaries

EvolutionIQ, a New York City-based company providing ML insights to large insurance carriers, overhauled its data platform, positioning the company for growth and strengthening developer morale. They adopted Dagster as their orchestration layer, moving from individual pipelines to a unified code base and achieving significant productivity gains. The team's adoption of Dagster was facilitated by a clear vision, hands-on champions, and incremental gains during the transition process. With the new data platform, EvolutionIQ reduced onboarding time for new clients from several months to less than a week, improved accountability, and enabled developers to focus on higher-level problems. The company has since moved to DataStack 2.1, which will integrate Dagster with Vertex AI for feature and tuning management.

Dagster, initially an open-source project by Elementl, aims to become a standard for managing data pipelines by addressing complexities in applications, operations, and enterprise needs. With the introduction of Dagster Cloud, a hosted orchestration service, users can focus on application complexity while offloading operational and enterprise complexities. Since its commercial release in December 2021, Dagster Cloud has been adopted by a range of customers and is now generally available. The company developed a pricing model that is consumption-based, charging for computation duration, encouraging efficient use without upfront costs. A simple self-serve plan and flexible hosting options, Serverless and Hybrid, cater to different needs, and an enterprise plan offers additional features such as unlimited deployments and enhanced support. Users can start with a free trial or explore enterprise options with account management support.

Data teams often spend a significant amount of time fixing broken assets and pipelines, with data engineers dedicating up to 80% of their day to debugging failures. This issue is particularly prevalent in large organizations where the root cause of an issue can lie in a source table managed by a different team. To tackle this problem, Dagster Labs has developed an orchestration platform designed to directly address productivity issues faced by data and ML engineers. The platform's critical role in debugging lies in its orchestrators, which schedule and order computations, making it easier to untangle what has happened and understand what was supposed to happen. Dagster's asset layer helps teams think about assets first and tasks when needed, providing a blueprint for analysis that guides human operators to debug faster. The platform's UI is designed to dramatically accelerate debugging by rapidly finding the root cause of failures and providing easy access to logs and metadata. By empowering organizations to build productive data platforms, Dagster aims to reduce the time spent troubleshooting issues and increase velocity.

Dagster Cloud, an enterprise orchestrator built for developer experience, has been released to general availability. It provides a managed cloud product with native branching, offloading operational burden from data teams, and best-in-class UI that serves as a single pane of glass for all stakeholders. The platform includes features such as Branch Deployments, which allow developers to create lightweight staging environments for testing and collaboration, and flexible, managed orchestration options like Serverless and Hybrid Dagster Cloud. This revolutionary development platform aims to empower all stakeholders with a seamless experience, enabling data teams to focus on business logic and deliver high-quality data assets. With its Enterprise capabilities and SOC 2 Type 1 certification, Dagster Cloud is poised to support companies of all sizes in achieving their goals.

Dagster, a data orchestrator, has reached version 1.0 after four years of development and 463 releases. The software-defined assets in Dagster are designed to manage critical data assets for teams, freeing them up to focus on building. The company held off declaring 1.0 until the core APIs were stable, which was achieved last year with versions of the computational abstractions. This year, the asset layer is now stable, allowing the team to declare 1.0. The release marks the work of over 200 people who contributed code and hundreds more who provided feedback and encouragement. Dagster's next steps include expanding its capabilities and exploring declarative asset orchestration.

The Open Core Business Model, articulated by Nick Schrock and the Elementl team, aims to make Dagster, an open-source project, a ubiquitous standard while generating revenue through its hosted SaaS platform, Dagster Cloud. The model emphasizes three types of complexity: application complexity (work done to structure code), operational complexity (how that code runs), and enterprise complexity (embedding data platforms in larger organizations). Dagster will maintain its core framework as open-source, with proprietary features and capabilities addressing the latter two complexities. The company's goal is to make Dagster accessible to everyone, drive adoption of the technology, and maximize revenue through a fair, sustainable pricing model.