Multiple Regions, Single Pane of Glass

A common challenge when building infrastructure-as-a-service products is providing highly available and isolated resources across multiple regions while maintaining a single pane of glass for end-users. To address this, WarpStream's architecture employs a "hub and spoke" model, where the primary region (us-east-1) serves as the central hub hosting the Aurora database that stores platform data, while separate regions act as spokes with their own isolated versions of the Metadata Store. This approach allows for regional independence while ensuring that critical path data is made available in all regions. To replicate this data across regions without creating inter-regional dependencies, WarpStream uses a "push-based" approach called "contexts," which involves building and updating metadata bundles (contexts) that are written to durable key-value stores in the spoke regions. These contexts are updated asynchronously to ensure fast propagation of changes while maintaining self-healing capabilities. The primary downside is the increased complexity of implementation, but WarpStream believes it was worth the effort to enable regional expansion.