Is Your RAG Consistent?

RAG, or Retrieval-Augmented Generation, combines information retrieval with generative AI to provide context-aware responses, yet inconsistencies in output can pose challenges, especially in regulated industries like finance and healthcare. These inconsistencies arise from the various components of a RAG stack, such as vector search and generative LLMs, which can behave differently with the same query, thus affecting the reliability of responses. To address this, the Open-RAG-Eval tool introduces a Consistency-Adjusted Index (CAI), which measures both the quality and stability of model outputs across multiple runs by factoring in the mean and standard deviation of various metrics, including BERTScore and ROUGE-L for semantic and lexical similarity. The CAI helps identify variability in generation behavior, providing insights into the stability and reliability of RAG systems. Through examples with different generation settings, the blog illustrates how the CAI can identify both subtle and significant differences in model performance, aiding in more dependable decision-making for RAG deployments in sensitive environments.