Introducing Legal RAG Bench

Legal RAG Bench is a new evaluation benchmark designed to assess the real-world performance of legal Retrieval-Augmented Generation (RAG) systems, emphasizing the importance of information retrieval over reasoning in these systems. The benchmark reveals that retrieval failures are often mistakenly attributed to reasoning errors, highlighting the Kanon 2 Embedder model's superior performance in retrieval accuracy compared to other models like Gemini 3.1 Pro and GPT-5.2. Legal RAG Bench uses a combination of complex questions and passages from the Judicial College of Victoria’s Criminal Charge Book, reflecting expert-level knowledge of Victorian criminal law, to evaluate models. The study shows that good retrieval can compensate for weak reasoning, but strong reasoning cannot offset poor retrieval. This benchmark encourages transparency and further research, providing a comprehensive and robust methodology for evaluating legal RAG systems.