Your KV cache benchmark is “hi hi hi”

Benchmarking KV cache offloading systems using synthetic inputs, such as repetitive token sequences, can lead to misleading performance evaluations because they do not accurately reflect real-world workloads. While benchmarks using simplistic inputs like repeated "hi" tokens may show excellent compression and transfer rates, they fail to capture the complexity of activation patterns, token diversity, and tensor value distributions found in actual deployments. The discrepancy between synthetic and realistic benchmarks becomes evident when comparing the token diversity of a typical generated document against a real-world document, such as those in medical or legal fields. The article highlights the importance of using representative inputs that mirror the diversity and structure of actual workloads to obtain trustworthy compression ratios and transfer sizes. It underscores the necessity for benchmarks to be based on realistic text to ensure accurate performance assessments of KV cache systems, cautioning against relying on synthetic benchmarks that do not resemble the data environments in which these systems operate.