January 2024 Summaries

On January 26th, a Gentrace customer reported a cross-site scripting (XSS) vulnerability found by a security consultant, which occurred due to unsanitized text rendered in OpenAI input and output blocks within the Gentrace UI. The issue was linked to the use of Mustache for interpolating content without automatic template sanitization. By January 28th, Gentrace addressed the vulnerability by escaping content before rendering and thoroughly scanning their production database, finding no malicious hits beyond the initial benign security test. To prevent future occurrences, Gentrace audited their codebase to ensure all user-generated content passed to React's dangerouslySetInnerHTML is sanitized with DOMPurify, and they implemented a react/no-danger eslint error to enforce safety checks during pull request reviews. With the fix in place and no users affected, no actions were recommended for customers, and Gentrace encouraged feedback on best practices while promoting their newsletter for updates and AI engineering insights.

AI hallucinations, where large language models (LLMs) produce incoherent or factually inaccurate responses, can be addressed through various iterative enhancements such as improved context, targeted prompts, fine-tuning, and post-processing. To test these enhancements, four strategies are proposed: automated fact comparison, automated specific checks, manual expert review, and end-user feedback. Automated fact comparison involves comparing AI outputs to expected values, while specific checks focus on particular types of hallucinations without needing expected values. Manual expert review, though slower, ensures accuracy by validating facts, and end-user feedback, despite being imperfect, provides a useful indicator of hallucination rates. Combining these methods can enhance the detection and correction of hallucinations in both development and production environments.

Retrieval-Augmented Generation (RAG) is a prominent pattern in modern large language model (LLM) development, combining information retrieval from data sources with response generation using LLMs. This approach, which includes various architectures such as keyword-based RAG and those using embeddings and Vector DBs, is applied in scenarios like internet searches, knowledge querying, and dataset summarization. While RAG facilitates the creation of MVPs for generative AI applications, developers face challenges in improving quality and reducing hallucinations. Best practices for testing RAG systems involve creating test data, conducting AI evaluations with models like GPT-4, and refining the evaluation process to distinguish between retrieval and generation system failures. Enhancements to the testing framework include evaluating specific stages of the RAG pipeline and breaking down fact evaluation into compliance and completeness assessments. These methodologies aim to systematically improve RAG system performance by identifying and addressing specific failure points.