MAEB: Evaluating Audio Embeddings at Scale

The Massive Audio Embedding Benchmark (MAEB) introduces a comprehensive evaluation framework for audio embeddings, incorporating 98 tasks across 100+ languages and baselines for over 50 models within the MTEB ecosystem. This initiative addresses the fragmentation in audio model evaluation by providing a unified platform, revealing that no single model excels universally. Large audio-language models such as LCO-Embedding-Omni-7B and Qwen2-Audio-7B show promise, although they exhibit specific weaknesses. The benchmark highlights significant challenges in multilingual audio processing, particularly for low-resource languages, and underscores the necessity for balanced training objectives that integrate acoustic and linguistic properties. MAEB's findings suggest that the quality of audio encoders correlates with the performance of Audio LLMs, offering a predictive measure for multimodal audio reasoning tasks. Despite its comprehensive scope, the benchmark is designed for practical usability, providing efficient evaluation paths for researchers with limited computational resources. The authors emphasize that MAEB is a foundational tool intended to evolve with contributions from the community, aiming to advance the field of audio embeddings.