August 2023 Summaries

Quarterly earnings reports are crucial for investors but can be challenging to analyze due to their unstructured format, prompting the need for advanced technological solutions. Traditional AI models, such as ChatGPT, face difficulties in processing these vast and visually complex documents, leading to the exploration of Retrieval Augmented Generation (RAG) pipelines, enhanced by tools like Unstructured's library. This approach leverages the computational power of large language models (LLMs) and the memory capabilities of vector databases to extract and analyze table data from earnings reports, using LangChain, ChromaDB, and OpenAI's models. The process involves converting tables into HTML format for better LLM interpretation, chunking information to fit LLM context windows, and embedding the data for query-based analysis. While effective, the method encounters challenges like missing table titles in HTML outputs and issues with table extraction models when faced with varied row background colors, prompting ongoing optimizations and further development efforts.

Unstructured.io provides a comprehensive toolkit for handling data ingestion and preprocessing, facilitating the setup of machine learning pipelines with a focus on Retrieval Augmented Generation (RAG). The guide details how to extract and preprocess data from Google Cloud Storage (GCS) using Unstructured's API and connectors, then upload it to a vector database like Pinecone. It covers essential steps such as enabling GCS access, running the Unstructured API, cleaning data by removing unclassified text elements, and structuring data for efficient processing by Large Language Models (LLMs). Additionally, the guide explains how to embed, store, and retrieve data chunks using Pinecone and OpenAI's API, thereby enhancing LLM performance by providing contextually relevant information. The document concludes by encouraging users to explore potential improvements and to join the community for further engagement.