Apache Kafka® Tiered Storage in Depth: How Writes and Metadata Flow

Apache Kafka Tiered Storage is a feature that allows data to be stored in a secondary cold storage system, reducing the load on the primary storage. This is achieved through KIP-405, which introduces an external store where most of the cluster's data is stored. The plugin, developed by Aiven, supports all three major cloud object stores and has been battle-tested in their Kafka fleet. Tiered Storage works by asynchronously tiering data to the external store, keeping both a local and remote copy for a subset of the data. Reads can come from either the external store or the local data, with preference given to the local one. The plugin uses chunk-based uploads to store data in the external store, which allows for efficient caching and prefetching mechanisms. Metadata is stored in an internal topic named `remote_log_metadata`, which is consumed by each broker to build up the latest remote segment state in memory. This feature allows users to build additional custom functionality on top of Kafka, making it a simpler and more cost-effective solution for large-scale data storage.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.