Streamlining large-scale data exports: The evolution of Labelbox’s Export System to a streamable architecture with fairness in place

Labelbox has transformed its data export system to a streamable architecture to improve efficiency and address challenges faced by users. Initially, the export system encountered issues such as a time-consuming chunk combining phase and stateful communications, which led to stale exports. To overcome these, Labelbox introduced a streamable export system using an iterator approach, allowing for the processing of large datasets without loading them entirely into memory. To ensure a smooth transition, the streamable backend was back-ported to existing methods, with feature flags via LaunchDarkly managing the rollout. Performance issues were further addressed by implementing a sophisticated queue management system with BullMQ, introducing a round-robin system for fair processing across organizations, and employing horizontal scaling to enhance throughput. This evolution has significantly improved data export processes, ensuring users can handle large-scale exports more effectively while maintaining a commitment to ongoing innovation and system refinement.

No tracked trend matches for this post yet.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.