Benchmarking Diskless Topics: Part 1

Aiven's blog post presents an open-source benchmark for Diskless Kafka (KIP-1150), which significantly reduces cloud costs by leveraging direct-to-S3 topic storage, potentially cutting infrastructure expenses by over 94%. The benchmark, run on a cluster of six m8g.4xlarge machines across three availability zones, demonstrated a 1 GiB/s input and 3 GiB/s output workload, maintaining stable performance with less than 30% CPU usage and achieving ~1.6 seconds P99 end-to-end latency. This new approach offers substantial savings on disk and cross-zone replication costs while maintaining the capability to run classic and Diskless topics side-by-side in the same cluster. Although Diskless Kafka introduces a slight increase in latency due to reliance on object storage, the trade-off is offset by substantial cost savings, making it an attractive option for future Kafka deployments. The post emphasizes the need for realistic and open-source benchmarks, as they provide a reliable data point for evaluating Diskless topics' performance and encourage community involvement in the ongoing development of Apache Kafka's Diskless architecture.