Open Evaluation

Vectara has developed Open RAG Eval, an open-source evaluation framework designed to enhance the assessment of Retrieval-Augmented Generation (RAG) systems by moving beyond traditional methods that rely on impractical "golden answers". Instead, it utilizes high-quality human judgments and language model-based assessments to evaluate relevance and helpfulness, making it accessible and user-friendly for developers, researchers, and product teams. Complementing this, the Open Evaluation website provides a platform to analyze and compare evaluation reports, offering insights into retrieval and generation metrics such as relevance, groundedness, factuality, and citations. As part of a broader initiative to foster a transparent and collaborative RAG ecosystem, these tools aim to facilitate actionable, continuous improvements in AI systems by engaging the community in the evaluation process.