September 2024 Summaries

Retrieval-Augmented Generation (RAG) systems face challenges in effectively retrieving relevant information from large document collections due to biases introduced by traditional single-step retrieval processes, which often focus on a limited number of documents. Ragie addresses these limitations by implementing a two-step retrieval process that incorporates document summarization, enhancing both the relevancy of retrieval and the breadth of document coverage. This system uses the Gemini 1.5 Flash model for summarizing documents, creating condensed versions stored in a Summary Index for high-level searches, while maintaining a Chunk Index for detailed retrievals. The two-tiered approach first identifies the most relevant documents using the Summary Index and then retrieves specific chunks from these documents in the Chunk Index, guided by a configurable parameter to ensure diversity in the search results. By integrating a final LLM re-ranking step, Ragie's method ensures the delivery of semantically relevant information from a wider set of documents, offering a more accurate and efficient RAG-based system for large-scale data sets.

Ragie employs a specialized approach to chunking tabular data extracted from various document formats, such as Word, PDF, CSVs, and spreadsheets, for improved semantic retrieval. The strategy aims to address common issues with naive table chunking, such as losing contextual information when chunks end in the middle of columns or rows, and the problem of invalid data in formats like XML, JSON, or YAML when data exceeds chunk size. The Ragie table chunker begins with a structured representation of the data and creates markdown-formatted table chunks, ensuring that table data remains associated with headers and rows are not split mid-record. If a table's size exceeds the chunk size, it is processed by row to fit within the limits, and for tables with many columns, the chunk size is adjusted to accommodate the data without excessive repetition of headers, ensuring that the data remains coherent and effective for hybrid search results.