May 2024 Summaries

The Align AI Research Review introduces an analysis framework called RGAR to systematically assess Retrieval Augmented Generation (RAG) systems. RAG is crucial in NLP for optimal retrieval methods and generating better responses. The RGAR framework considers the Target, Dataset, and Metric comprehensively. It provides relevance, accuracy, and faithfulness by encompassing both potential output and ground truth pairings. The evaluation process includes three key questions: what should be the Evaluation Target, how should the Evaluation Dataset be assessed, and how should the Evaluation Metric be quantified? Retrieval metrics focus on relevance, precision, diversity, and reliability, while generation metrics emphasize coherence, relevance, fluency, and alignment with human perception. The research also discusses additional requirements such as latency, diversity, noise robustness, negative rejection, and counterfactual robustness.

The Align AI Research Review discusses Generative AI technologies and their limitations based on training data. It introduces Retrieval Augmented Generation (RAG) as an alternative solution to these challenges, integrating external knowledge sources into Language Models (LLMs). Meta's paper proposes a novel model-agnostic framework called LLM-augmented retrieval, which enhances the performance of existing retriever models by improving document embeddings through LLM augmentation. The framework involves generating synthetic relevant queries and titles for original documents, splitting long documents into passages, and adapting retrieval frameworks for varied model architectures. While this approach offers improvements in information retrieval tasks, it also presents challenges such as increased computational demand and potential vulnerability to errors or biases from large language models.

Self-Retrieval is a novel architecture for end-to-end information retrieval that utilizes large language models (LLMs). It improves the efficacy of downstream applications and outperforms previous retrieval methods. The proposed system integrates LLMs into storing the corpus to be retrieved by internalizing the documents and creating a natural language index. Self-Retrieval consists of three steps: indexing, retrieval, and self-assessment. This design allows a single LLM to entirely execute the retrieval task. Compared to sparse and dense retrieval baselines, self-retrieval shows an average improvement of 11% in MRR@5. Further investigation is needed to understand the scaling law heading the link between document size and model parameters.