February 2024 Summaries

Klick Health, the world's largest independent commercialization partner for healthcare and life sciences, is pioneering AI-powered applications to accelerate growth and improve experiences and outcomes for patients and consumers. As a Data Science Team Leader at Klick Consulting, Peter Leimbigler leads a team that helps define and solve complex problems in the healthcare and life sciences space across various clients. The company has established a generative AI center of excellence to support its own internal operations and client-facing projects, and is exploring ways to use large language models (LLMs) effectively and responsibly. LLMs have shown promise in speeding up drug discovery and development, supporting clinical trials, augmenting doctor-patient interactions, and personalizing patient experiences. However, governing LLM behavior poses unique challenges, particularly in the heavily regulated area of healthcare. Klick has adopted tools like Phoenix for LLM observability to address these challenges and ensure that its AI applications deliver tangible business impact. The company prioritizes outcomes over optics, focusing on reproducibility, transparency, and clarity of data narratives to make meaningful work and collaborate with clients to achieve business results.

Ragas provides a robust workflow for building and evaluating RAG pipelines, utilizing open-source libraries such as Ragas, Arize AI's Phoenix, and LlamaIndex. The pipeline involves generating synthetic test data using Ragas, building a simple RAG application with LlamaIndex, launching Phoenix to collect traces and spans, and evaluating the performance of the LLM application using Ragas. Additionally, Phoenix provides visualization tools for analyzing embedded queries and retrieved documents, allowing developers to identify areas of poor performance and gain insights into their application's behavior. By combining Ragas and Phoenix, developers can create a comprehensive evaluation framework for their RAG pipelines, ensuring high-quality responses and efficient model development.

In the evaluation of retrieval-augmented generation (RAG), the focus is often on the retrieval stage while the generation phase receives less attention. A series of tests were conducted to assess how different models handle the generation phase, and it was found that Anthropic's Claude outperformed OpenAI's GPT-4 in generating responses. This outcome was unexpected as GPT-4 usually has a strong lead in evaluations. The verbosity of Claude's responses seemed to support accuracy, as the model "thought out loud" to reach conclusions. When prompted to explain itself before answering questions, GPT-4's accuracy improved dramatically, resulting in perfect responses. This raises the question of whether verbosity is a feature or a flaw. Verbose responses may enable models to reinforce correct answers by generating context that enhances understanding. The tests covered various generation challenges beyond straightforward fact retrieval and showed that prompt design plays a significant role in improving response accuracy. For applications that synthesize data, model evaluations should consider generation accuracy alongside retrieval.

The paper "RAG vs Fine-Tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture" explores the use of retrieval augmented generation (RAG) and fine-tuning in large language models. It presents a comparison between RAG and fine-tuning for generating question-answer pairs using high-quality data from various sources. The authors discuss the benefits and drawbacks of both approaches, emphasizing that RAG is effective for tasks where data is contextually relevant, while fine-tuning provides precise output but has a higher cost. They also highlight the importance of using high-quality data sets for fine-tuning and suggest that smaller language models may be more efficient in certain cases. The paper concludes by stating that RAG shows promising results for integrating high-quality QA pairs, but further research is needed to determine its effectiveness in specific use cases.