November 2023 Summaries

The Hallucination Index benchmark evaluates the performance of popular Large Language Models (LLMs) in generating correct and contextually relevant text, with a focus on detecting model hallucinations. The index is designed to help teams select the right LLM for their project and use case by providing a framework to address the variability and nuance that comes with generative AI. It uses seven rigorous benchmarking datasets to evaluate each LLM's performance across three task types: Question & Answer without Retrieval (RAG), Question & Answer with RAG, and Long-form Text Generation. The index ranks LLMs by task type, providing insights into the strengths and weaknesses of each model in addressing hallucinations. By utilizing a combination of quantitative metrics, such as Correctness and Context Adherence, and human evaluations, the Hallucination Index offers a comprehensive evaluation metric for detecting hallucinations in generative AI applications.

OpenAI's recent Dev Day event brought exciting announcements and updates that are likely to shape the future of AI development. The company unveiled the GPT-4 Turbo model, which boasts a 128K context window, allowing it to process large amounts of text in a single prompt. This new model is more capable than its predecessor, with improved performance on tasks requiring precise instructions. OpenAI also introduced the Assistants API, enabling developers to build agent-like AI applications similar to character.ai. The company's commitment to customer protection is evident with the introduction of Copyright Shield, which defends customers against legal claims related to copyright infringement. Additionally, OpenAI has doubled the limit of tokens per minute for all paying GPT-4 customers, allowing developers to scale their applications more efficiently. The company has also reduced pricing for various models, making AI more affordable than ever. Furthermore, OpenAI released a new major version of its python SDK, which comes with improved features and enhanced functionality. Overall, these updates empower organizations to create more sophisticated and cost-effective AI-driven solutions, giving them a competitive edge in the fast-evolving AI landscape.

The Biden administration has issued a groundbreaking executive order to regulate AI, focusing on safety, privacy, innovation, and global leadership. The order sets new standards for AI safety and security, building upon previous initiatives with 15 leading companies committed to ensuring the safe development of AI technologies. To address potential risks, the administration is promoting effective Red-teaming, a process that involves simulating adversarial scenarios to discover vulnerabilities in AI systems. Organizations must develop frameworks to evaluate and reduce model harm, ensure transparency and sharing of safety test results, prioritize privacy-preserving techniques, and address algorithmic discrimination. The executive order marks an exciting era for trustworthy AI innovation, where progress and the well-being of Americans are at the forefront.