May 2025 Summaries

Vanta, a leader in trust management and compliance automation, adopted Dagster as their modern data orchestration platform to eliminate data bottlenecks. By doing so, they drastically reduced data delays, streamlined on-call operations, and significantly expanded self-service capabilities for developers, data engineers, and analysts. The adoption of Dagster enabled Vanta to achieve a unified programming model, central visibility, and the ability to integrate with their existing data stack while providing a foundation for future growth. This resulted in improved data freshness, reduced troubleshooting time, minimized escalations, and reduced time to new data sources and integrations. Additionally, Dagster empowered data analysts to independently manage most incidents during on-call shifts, decreasing off-hours escalations. The team also reported increased productivity and efficiency, with junior developers leading important initiatives and a significant reduction in spend and reliance on legacy tools. Vanta plans to expand the possibilities enabled by Dagster, including automated testing in CI, deeper integration with Sigma, and OpenAI.

Data orchestration platforms offer a more robust foundation for scalable data engineering than traditional iPaaS solutions like integration platforms. These modern platforms address limitations by providing code-first, developer-friendly interfaces that integrate with existing workflows, end-to-end observability, and asset-centric architecture that focuses on data products rather than just connections. They provide advantages such as version control integration, built-in testing capabilities, clear lineage tracking, automatic monitoring and alerting, and can handle complex workflows by providing reusable connectors for legacy systems. Data orchestration platforms are better suited for building scalable, maintainable platforms with developer experience tools that enhance productivity, observability, reusability, and testing capabilities.

StashAway, a digital wealth management platform operating in five global markets, has unified its global data platform with Dagster, an open-source automation and orchestration system. By leveraging Dagster's capabilities, StashAway has increased developer velocity by 50%, improved reliability and stability, and achieved end-to-end lineage across multiple Kubernetes clusters. The company has also adopted software-defined assets, which have enabled them to streamline their workflows, reduce costs, and improve data quality. With Dagster at the center of its data stack, StashAway is able to manage its multi-region infrastructure with ease, ensuring daily data readiness through proactive data quality checks and automatic retries.

In the Vibe Coding Survival Guide, Dennis Hume shares insights and best practices for effective AI-assisted development with frameworks like Dagster. To get the most out of coding alongside an AI agent, start small, think in steps, test each step, stay opinionated about tools, be mindful of types, and remember that you are the engineer. By following these guidelines, you can build more scalable data platforms, maintain high standards, write understandable code, and preserve long-term maintainability while leveraging the power of AI for productivity and efficiency.

Dagster Pipes, a standardized interface for launching code in external environments with minimal additional dependencies, is now available for TypeScript, Rust, and Java. This expansion allows data teams to work seamlessly across diverse technology stacks, providing full visibility through parameter passing, streaming logs, and structured metadata. With the new implementations, teams can leverage Dagster Pipes in more ecosystems, standardize orchestration across their stack, and integrate with popular languages such as Node.js for backend teams, Rust for high-performance data processing, and Java for enterprise-scale applications. The new features enable standardized orchestration capabilities, great performance, strong memory safety guarantees, compatibility with JVM-based data ecosystems, and support for large-scale, mission-critical workflows. Users can now easily integrate Dagster Pipes into their TypeScript, Rust, or Java projects, following the documentation provided for each implementation.

Dagster's Asset Selection Framework is a new feature that enables users to quickly find, understand, and act on the precise subset of their data platform that matters right now. This framework cuts through complexity by providing improved selection syntax and asset favoriting capabilities, allowing users to drill down to exactly what they need in seconds. It also helps users understand relationships between critical assets and take action by materializing and monitoring the assets that matter. The framework is designed to make Dagster the definitive source of truth for understanding and managing data platforms, regardless of their complexity. With this feature, users can solve domain-specific problems, such as searching for exactly what they want in a complex system, and model their data platform to conform to how they do business while staying in compliance within the same workstream. The framework also supports logical operators, parenthetical grouping, and specialized tokens that help traverse asset lineage up or downstream. Users can get started with this feature by upgrading Dagster to version 1.10.5, reviewing documentation, tagging and annotating assets, experimenting with the syntax, and following best practices.