Measuring Hallucinations in RAG Systems

Vectara has released an open-source Hallucination Evaluation Model (HEM) that provides a FICO-like score for grading how often generative LLMs hallucinate in Retrieval Augmented Generation (RAG) systems. The model helps mitigate the risks of hallucinations, such as large errors or introducing biases due to training data, by evaluating the trustworthiness of RAG systems and identifying which LLMs are best suited for specific use cases. The HEM provides a scorecard that compares various models, including GPT4, GPT3.5, and others, on their hallucination rates, accuracy, and summary length, allowing users to make informed decisions about their generative AI adoption.