August 2024 Summaries

IBM Watson X offers powerful tools to build and test Generative AI applications, which can become complex stacks of tools and workflows. To understand and evaluate these applications, a powerful evaluation platform is needed. In this demo, Galileo's platform will be used alongside the IBM Watson X SDK to measure RAG performance, detect hallucinations, and iterate through numerous prompts and Large Language Models.

The text discusses the evaluation of Large Language Models (LLMs) in Retrieval-Augmented Generation (RAG) systems. It highlights the importance of comprehensively assessing LLMs for RAG tasks, considering various dimensions such as instructional purposes, context length, domain, and information integration. The text also introduces ChainPoll, a high-efficacy method for LLM hallucination detection, which leverages chain-of-thought prompting and polling to provide accurate and detailed explanations. ChainPoll is compared to other evaluation metrics like RAGAS (Retrieval Augmented Generation Assessment) and TruLens, highlighting its advantages in terms of accuracy, cost-effectiveness, and efficiency. The text also discusses the limitations of existing benchmarks, such as ChatRAG-Bench, and proposes a new approach called CRAG (Comprehensive RAG Benchmark), which aims to comprehensively evaluate LLMs for RAG tasks. Additionally, the text provides guidance on how to evaluate RAG systems, including defining clear objectives, selecting appropriate benchmarks, conducting comprehensive testing, and incorporating human evaluations.

The rapidly evolving software development landscape introduces agentic workflows that pose challenges in ensuring reliability and predictability due to their non-deterministic nature. To overcome these hurdles, teams require a specialized stack that guarantees precision, safety, and reliability.

There is still much to learn about large language models (LLMs) and the various types of hallucinations they can produce, with many teams having been working on LLM applications for over a year. Building production-ready apps using LLMs remains challenging due to their complexity, but a group of AI builders has compiled an in-depth guide to help streamline RAG workflows and optimize performance. The guide provides actionable tips such as reranking for relevance, efficient embedding, implementing query classification, selecting appropriate retrieval methods, and optimizing chunking. Additionally, multimodal models are gaining popularity, but they can also be prone to hallucinations, which can be mitigated by understanding the different types of hallucinations across modalities, detecting and evaluating extrinsic hallucinations, and learning how to evaluate generative AI initiatives effectively. The GenAI infra stack is constantly evolving, with efforts to codeify the various components powering the AI revolution.

Open-source models have caught up with private ones in terms of performance, particularly for Retrieval Augmented Generation (RAG) tasks. The cost of vector databases can be eliminated with long context RAG, offering a more affordable alternative. Open-source LLMs provide flexibility and customization, allowing organizations to fine-tune models for specific needs. They also offer lower initial and ongoing costs, improved security through greater control over the software, and enhanced compliance. Gemini 1.5 Flash is an excellent balance between performance and cost, achieving great context adherence scores at a fraction of the cost of other models like GPT-4o. By selecting the right model for your use case, enterprises can deliver high value at an affordable budget.