🏟️ Smol AI WorldCup: A 5-Axis Benchmark That Reveals What Small Language Models Can Really Do
Blog post from HuggingFace
The Smol AI WorldCup introduces a novel benchmark for evaluating small language models, focusing on five key axes: size, honesty, intelligence, speed, and efficiency. This benchmark addresses the limitations of traditional evaluations by considering the deployment realities of edge AI, where performance per resource unit is crucial. The SHIFT framework and WorldCup Score (WCS) provide an integrated evaluation system, revealing that smaller models can often outperform larger ones in efficiency and quality. Notably, a 4B model surpasses an 8B model in quality at a fraction of the RAM, and a 1.5GB Mixture-of-Experts model achieves similar performance to much larger dense models. The evaluation methodology, developed in collaboration with the FINAL Bench research team, includes a rotating question set to ensure long-term benchmark integrity and invites ongoing community participation.