Apache Kafka® Deserves Topic Types

Kafka, initially developed by LinkedIn for on-premise data centers, utilized local disk storage for data durability through replication, creating an abstraction with topics and partitions that required no naming for storage methods. Over time, as cloud computing became prevalent, Kafka evolved to support tiered storage, allowing older data to be offloaded to cloud object storage, leading to the emergence of tiered and remote-enabled topics. Recognizing the shift towards cloud-based storage, companies developed Kafka-compatible systems using object storage for all topic data, prompting initiatives like KIP-1150 to modernize Kafka with diskless topics and introduce explicit naming for "classic" and "diskless" topics. This evolution in Kafka's topic types, alongside discussions on potential future developments like bridging operational and analytical data, reflects the ongoing need for clearer terminology and possibly a new config, topic.type, to explicitly define topic types. The Kafka community is encouraged to explore making topic type an explicit concept, enhancing clarity in the evolving landscape of Kafka deployments.