Case Study: Incrementally Adopting Dagster at Mapbox

The Geodata teams at Mapbox are responsible for updating the map of the world that powers their developer data products and services. They process a large amount of data using various pipelines and flows to create maps, road network data, point of interest datasets, and search indices. However, they found that working with Airflow was painful and costly, particularly in terms of development time and infrastructure costs. To address this, they started using Dagster, which allows for incremental adoption on top of their existing Airflow installation. With Dagster, they can write pipelines using clean abstractions, compile them into Airflow DAGs, and deploy them on their production infrastructure. This has improved their developer experience, reduced costs, and speeded up the delivery of new data products. The team was able to reduce the time it takes to conflate all sources in a region from days or weeks to 1-2 hours, and testing is now so much less costly that it's become a major benefit. Dagster has also provided a path for incremental adoption without requiring a scratch rewrite of their existing pipeline codebase.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.