Cloud Topics: the metastore

The text explains the development of a custom metadata storage solution, referred to as the metastore, for managing data in Redpanda's Cloud Topics, which optimizes the process of mapping Kafka offsets to object storage locations and tracking metadata such as leader term boundaries and compaction states. The metastore is built using a key-value store inspired by LevelDB and RocksDB, featuring a Log-Structured Merge-tree (LSM) with pluggable persistence layers for durability and scalability, leveraging object storage and Raft for replication and failover. This design enables efficient disaster recovery and the use of read replicas, with a recovery point objective defined by a ten-minute flush interval. The metastore's architecture supports the Cloud Topics use case with the potential for further optimization and application in future Redpanda projects, and users can explore this technology with an Enterprise trial of Redpanda 26.1.