July 2024 Summaries

With Arize Phoenix, developers can quickly iterate during development and experiment phases of building an LLM application by running it locally, but for production or collaboration, they need to deploy Phoenix. The tool offers data persistence options via SQLite or PostgreSQL databases, enabling the ability to persist application telemetry data and collaborate with colleagues. Phoenix is an open source tool that helps engineers trace, evaluate, and iterate on generative AI applications, providing features such as logging traces, persisting datasets, running experiments, and sharing insights with colleagues.

Developing an AI assistant tailored for data scientists and AI engineers called Arize Copilot involved numerous challenges and valuable lessons about developing with LLMs. The tool is designed to assist users in troubleshooting and improving their models and applications through an agentic workflow, leveraging the Completions API from OpenAI for better control over state management. Lessons learned include managing state effectively, handling model swaps cautiously, using prompt templates with clear instructions and guidelines, incorporating data into prompts in a structured format, configuring function calls explicitly, implementing streaming efficiently, focusing on user experience, and utilizing testing strategies with datasets and automated workflows.

LLM instrumentation is crucial for achieving performance and reliability in large language models. LLM tracing helps track down issues such as application latency, token usage, and runtime exceptions, providing detailed insights into the model's behavior. OpenTelemetry (OTel) enhances tracing by offering standardized data collection and integration with various LLM frameworks. However, OTel may not be suitable for all LLM applications, and manual instrumentation or additional frameworks like OpenInference are necessary to properly instrument an LLM app. Automatic instrumentation offers comprehensive coverage but requires less control over the details of what is traced, while manual instrumentation provides flexible control but demands more effort to implement. Various methods for manual instrumentation, such as using decorators, the `with` clause, and starting spans directly, can be employed to customize tracing in LLM applications.

DSPy Assertions is a programming construct that expresses computational constraints for language model (LM) pipelines, integrated into the recent DSPy programming model. The researchers propose strategies to use assertions at inference time for automatic self-refinement with LMs. They found that LM Assertions improve not only compliance with imposed rules but also downstream task performance, passing constraints up to 164% more often and generating up to 37% more higher-quality responses.

The use of function calling in large language models (LLMs) enables developers to connect LLMs with external tools and APIs, enhancing their utility at specific tasks. However, evaluating the performance of function calls in LLM pipelines is becoming increasingly critical as more applications are deployed into production. Evaluating function calls involves examining each step of the process, including routing, parameter extraction, and function generation. An open source library called Phoenix offers a built-in evaluator to measure the performance of function calling within major LLMs, providing a tool for tracing and evaluation.

Arize AI Copilot is an innovative AI Assistant for AI that provides an intelligent, integrated solution for model and application improvement. It reduces manual effort, accelerates troubleshooting, and offers advanced tools for LLM development and data curation. Key features include versatile skill set, advanced LLM development, prompt optimization, and powerful data curation. Arize Copilot revolutionizes the workflow by integrating traditional processes and automating complex tasks, making it an invaluable assistant for data scientists and AI engineers.

The newly released instrumentation module in LlamaIndex's latest version (v0.10.20) offers a structured and flexible approach to event and span management, replacing the legacy callbacks module. The new module introduces several core components, including Event, EventHandler, Span, SpanHandler, and Dispatcher, designed to optimize monitoring and management of LLM applications. These components work together to provide granular views of application operation, pinpoint notable events, and track sequences of operations within complex applications. By integrating with Arize Phoenix, developers can fine-tune performance, diagnose issues, and enhance the overall functionality of their LLM applications, achieving deeper insights and more effective management of their systems.