Under the hood: Redpanda Cloud Topics architecture

Cloud Topics, now in general availability with Redpanda Streaming 26.1, introduces an innovative architecture that uses object storage for data while storing metadata locally, optimizing for cost, latency, and performance. This system allows for the separation of data and metadata storage, bypassing traditional cross-AZ networking costs associated with Raft replication. Data is initially batched in memory before being uploaded to cloud storage as L0 files, which are later optimized into L1 files for efficient reading by a background process called the Reconciler. The use of placeholders in Raft logs ensures that transaction and idempotency guarantees are maintained. This new replication mechanism is designed to slash costs by leveraging object storage, and Redpanda provides resources for users to explore and provide feedback on Cloud Topics.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.