Introducing Dagster Pipes

Dagster Pipes is a protocol introduced by Dagster that standardizes the process of invoking and retrieving structured results from external computations, allowing for lower complexity burdens for both infrastructure engineers and stakeholders writing business logic. It decouples orchestration and execution in Dagster while retaining orchestration and metadata-driven features, enabling users to launch code in external environments with minimal additional dependencies and get parameter passing, streaming logging, and structured metadata back into Dagster's lineage, observability, and data quality tools. Pipes is designed for teams that have invested countless engineer-hours into their existing data platform infrastructure, allowing them to adopt capabilities on top of software that works without a risky, costly rip-and-replace approach. The protocol is pluggable on a per-capability basis, supporting first-class multi-language support in Dagster and enabling users to write business logic and transformations in any programming language, including Rust, Julia, R, Java, and others. With Dagster Pipes, users can bring every computation that produces data into the fold, apply observability, lineage, data quality, consumption management, and great developer workflow across the entire platform.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.