Scaling Dagster’s DAG Visualization to Handle Tens of Thousands of Assets

Dagster, an orchestrator for data assets, is witnessing rapid adoption across industries and use cases. Currently, users build complex pipelines with thousands of assets, which poses challenges on the front-end rendering enormous graphs. To address this, Dagster's UI team has implemented several optimizations to improve performance. These include virtualizing asset nodes and edges to reduce rendering lag, batching data loading into chunks, and caching layout computations using IndexedDB. Additionally, they have switched from a complex layout algorithm to a simpler one that can be more efficiently rendered, resulting in a significant reduction of rendering time from minutes to under five seconds for large asset graphs with 2,000 assets. The team is also exploring new UX features, such as improved navigation and group views, to further enhance the user experience.