Compute:Arena / Measuring Local Inference Across Models, Quants, Chips, and Runtimes

Compute:Arena is an open-source public leaderboard and command-line benchmarking tool designed to measure local LLM inference performance across combinations of model files, quantization formats, hardware chips, and runtimes. It records standardized prefill throughput across prompts from 128 to 16,384 tokens and decode throughput over 128 tokens, using warmups and three repetitions to make submitted results comparable at each workload size. The platform currently supports BaseRT for .base model bundles and llama.cpp’s llama-bench for .gguf files, while identifying models and runtime executables through cryptographic hashes and distinguishing quantization variants by format. Each benchmark produces a signed local report containing raw timings, chip-detection details, model and executable integrity checks, and telemetry such as temperature, power state, memory pressure, swap activity, and GPU snapshots. Although signatures protect reports from later alteration, the project notes that they cannot verify the truthfulness of machine-reported data and that results from different runtimes are not necessarily exactly equivalent because their benchmark behavior differs. Users can run benchmarks locally, inspect the JSON before submission, and contribute results, issues, or code through the Apache-2.0-licensed project.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.