August 2024 Summaries

This tutorial guides users through setting up an image classification experiment using Phoenix, a multi-modal evaluation and tracing platform. The process involves uploading a dataset, creating an experiment to classify the images, and evaluating the model's accuracy. OpenAI's GPT-4o-mini model is used for the classification task. Users are required to have an OpenAI API key ready and install necessary dependencies before connecting to Phoenix. The dataset is loaded from Hugging Face, converted to base64 encoded strings, and then uploaded to Phoenix. After defining the experiment task using OpenAI's GPT-4o-mini model, evaluators are set up to compare the model's output with the expected labels. Finally, the experiment is run, and users can modify their code and re-run the experiment for further evaluation.

The State of AI Engineering survey reveals that industries are rapidly adopting large language models (LLMs) for various applications such as summarizing medical research, navigating complex case law, and enhancing customer experiences. Over half of the surveyed AI teams plan to deploy small language models in production within the next 12 months. The most common use cases include chatbots, code generation, summarization, and structured data extraction. Privacy concerns, accuracy of responses, and hallucinations are identified as the top implementation barriers for LLMs. Prompt engineering is widely used by AI teams, with nearly one in five relying on LLM observability tools to evaluate and trace generative AI applications. Developers and AI teams show a preference for both open-source and proprietary models, with a slight increase in interest in third-party cloud-hosted options. The majority of respondents are neutral or against more regulation of AI, while Python remains the preferred language for serving LLMs.

Observability is crucial in autonomous AI agents as it allows monitoring and evaluation of their performance. CrewAI is an open-source agent framework that enables the creation and design of personalized employees to automate tasks or run businesses. Key concepts include agents, tasks, tools, processes, crews, and pipelines. Observability can be set up using Arize Phoenix for real-time insights into AI agents' activities and performance. CrewAI offers enhanced visibility, streamlined task management, collaborative intelligence, scalability, and customization options.

In the Arize Release Notes on Aug 23, 2024, users can now create spaces programmatically using graphQL. Online evals have been updated with support for three new LLM integrations: Azure OpenAI, Bedrock, and Vertex / Gemini. Event-based Snowflake jobs are also introduced, allowing users to trigger Snowflake queries via graphQL. The Python SDK v7.20.1 includes enhancements such as delayed tags for stream logging, experiment eval metadata, and ingesting data to Arize using space_id. Additionally, new content has been published on topics like tracing LLM applications, LlamaIndex workflows, types of LLM guardrails, annotations for human feedback, evaluating alignment and vulnerabilities in LLMs-as-judges, Flipkart's use of generative AI, and Atropos Health leveraging LLM observability.

Bazaarvoice, a leading platform for user-generated content and social commerce, has successfully deployed an LLM app after navigating through challenges related to data quality and education. The first challenge was ensuring the data used in retrieval augmented generation (RAG) was clean and accurate, especially when it came to business-specific data. The second challenge involved educating employees about AI's capabilities and limitations. Bazaarvoice has found that AI is transforming its business by improving content quality for clients and enhancing the user experience through generative AI applications like a content coach.

Combining Phoenix with Haystack enables effortless tracing, enhanced debugging, and comprehensive evaluations for LLM applications and search systems. With just a single line of code, Phoenix provides deep insights into application behavior through tracing, allowing developers to pinpoint issues quickly. To set up a basic RAG application using Haystack and Phoenix, install the necessary libraries, launch a local Phoenix instance, connect it to your application, and add the Haystack auto-instrumentor to generate telemetry. Initialize your Haystack environment by setting up a document store, retriever, and reader, and build a RAG pipeline with components such as a retriever, prompt builder, and LLM. Finally, call the Haystack pipeline with a question and view the resulting LLM traces in Phoenix for further analysis.

This paper evaluates the performance of various LLMs acting as judges on a TriviaQA benchmark. The researchers assess the alignment between the judge models' outputs and human annotations, finding that only the best-performing models (GPT-4, Turbo, and Llama 3 7 B) achieve high alignment with humans. The study highlights the importance of using top-performing models for evaluating LLMs as judges. The results also show that larger models tend to perform better than smaller ones, but the difference in performance is not always significant. Additionally, the paper finds that prompt optimization and handling under-specified answers can improve the performance of LLM judges. However, it's essential to note that this study is conducted in a controlled environment and might not generalize well to real-world use cases. The authors recommend using Cohen's Kappa as a metric for evaluating alignment between human evaluators and LLM judges, which accounts for agreement by chance.

Phoenix is an AI development platform that allows users to collect human feedback on their Large Language Model (LLM) applications, making it easier to evaluate and improve these models. The platform provides a robust system for capturing and cataloging human annotations, which can be added via the UI or through SDKs or API. Annotations can be used to create datasets, log user feedback in real-time, and filter spans and traces in the UI or programmatically. Phoenix also integrates with its Datasets feature, allowing users to fine-tune their models using annotated data. The platform enables a new system of collecting human feedback en masse through reinforcement learning from human feedback (RLHF), popularized by the rise of LLM-based evaluations. With Phoenix, users can now log human feedback into their applications, combining automated metrics with human insights to create models that not only perform well but also resonate with users.

Atropos Health is working to close the evidence gap by making observational studies easily accessible for physicians. The company's Principal Data Scientist, Rebecca Hyde, has over ten years of experience in data science and public health. They have developed AutoSummary, an LLM-based prompting tool that produces summaries of these studies, which physicians can edit. To measure the performance of AutoSummary at scale, they are setting up a monitoring framework using LLM Observability. Hyde spoke about her experiences with Arize:Observe, where she learned more about LLMs and their use cases.

The text discusses example notebooks available from OpenInference on various topics such as RAG pipelines and building fallbacks with conditional routing using Haystack, Groq, and other libraries. It also introduces a tutorial on instrumentation for LLM applications, covering key frameworks like OpenTelemetry (Otel) and OpenInference, along with the pros and cons of automatic and manual instrumentation. The tutorial demonstrates three methods for setting up manual instrumentation: using decorators, the `with` clause, and starting spans directly. Additionally, it mentions new content in video tutorials, paper readings, ebooks, self-guided learning modules, and technical posts.

Flipkart has leveraged generative AI to support its 600 million users, with a primary goal of improving customer experience and scaling their business. The company's Head of Applied AI, Anu Trivedi, discussed the challenges of measuring success in product development and how Flipkart is using generative AI to create conversational commerce opportunities. With the help of Arize, a partner that provides traceability, Flipkart has gained the ability to stitch together various metrics and create a storyline to improve their product. The company's experience with generative AI has been a learning process, highlighting the importance of understanding customer bases and finding the right route for solution implementation. By leveraging generative AI, Flipkart aims to make a significant impact on its business, particularly in terms of ROI and GMV.

LlamaIndex has released a new approach to easily create agents called Workflows, which use an event-based architecture instead of traditional pipelines or chains. This new approach brings new considerations for developers and questions on how to evaluate and improve these systems. Workflows are an orchestration approach that lets you define agents as a series of Steps, each representing a component of your application. This event-driven architecture gives more freedom to applications to jump around and allows steps to be self-contained, making it easier to handle intricate flows and dynamic applications. Workflows are great at handling complicated agents, especially those that loop back to previous steps, but may add unnecessary complexity to linear applications. To visualize the paths taken by your Workflows, you can use Arize Phoenix, which provides an integration with LlamaIndex that allows you to easily visualize step-by-step invocations without adding extensive logging code. Workflows can also be evaluated using a similar approach as evaluating any agent, breaking down the process into tracing visibility, creating test cases, breaking components into discrete parts, defining evaluation criteria, running test cases, and iterating on your app.

Llama 3 is a large language model developed by Meta AI that has been trained on diverse data sources with an emphasis on multilingual content. The flagship model of the Llama series, Llama 3-70B, boasts impressive performance in various benchmarks and tasks, including coding, reasoning, and proficiency exams. It also supports eight languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai. One of the key features of Llama 3 is its long context window capability, which allows it to retrieve information from large documents effectively. However, it has been found to be more susceptible to prompt injection compared to other models like GPT-4 and Gemini pro. Llama 3's open-source nature makes it accessible for developers and researchers to fine-tune the model according to their needs. Meta AI also released a guardrail model, which can be used as a small model to detect and prevent potential prompt injections or undesired token generations. Overall, Llama 3 showcases significant advancements in large language models and contributes to the growing field of open-source AI development.

Arize AI introduces EU data residency support for all users, allowing them to host their data within the European Union while adhering to local data protection laws such as GDPR. This feature is particularly beneficial for sectors like finance and healthcare where data storage regulations are stringent. Arize is also SOC 2 Type II, HIPAA compliant, and has achieved PCI DSS 4.0 certification. For more information on Arize AI's commitment to compliance and data privacy requirements globally, visit the Arize Trust Center.

This research explores the effectiveness of using Large Language Models (LLMs) as a judge to evaluate SQL generation, a key application of LLMs that has garnered significant interest. The study finds promising results with F1 scores between 0.70 and 0.76 using OpenAI's GPT-4 Turbo, but also identifies challenges, including false positives due to incorrect schema interpretation or assumptions about data. Including relevant schema information in the evaluation prompt can significantly reduce false positives, while finding the right amount and type of schema information is crucial for optimizing performance. The approach shows promise as a quick and effective tool for assessing AI-generated SQL queries, providing a more nuanced evaluation than simple data matching.