February 2025 Summaries

AI engineers face the challenge of bridging the gap between development and production while ensuring high performance across diverse AI model types. Traditionally, these phases are treated as separate entities, but in reality, they are deeply interconnected. Arize's unified AI observability and evaluation platform bridges this gap by providing end-to-end observability, evaluation, and troubleshooting capabilities across all AI model types, enabling teams to develop with confidence, monitor and debug production applications, use online production data for continuous experimentation and iterative development, and connect development and production in a single feedback loop. Arize supports the full spectrum of AI-powered systems and applications, including generative AI, computer vision, and machine learning models, providing a single pane of glass to monitor, evaluate, and iterate across LLMs, CV, and ML models alike.

Memory in Large Language Model (LLM) applications refers to any mechanism by which an application stores and retrieves information for future use. It encompasses two main types of state: persisted state, stored in external databases or durable storage systems, and in-application state, retained only during the active session and disappearing when the application restarts. LLM models are inherently stateless, processing each query as a standalone task based solely on the current input. However, for applications requiring context continuity, managing memory and state is essential to deliver consistent, coherent, and efficient user experiences. Effective state management balances the need for long-term context with the costs of storage and retrieval. Strategies include tiering memory to prioritize what's most important, using specialized entities or memory variables, semantic switches, and advanced write and read operations to optimize performance and cost. Evaluating state management is critical to understanding its impact on application performance, and techniques such as running LLMs as judges, incorporating human annotations, and measuring persisted state usage can help refine state management systems. As applications become increasingly complex, the balance between simplicity and intelligence in state management will be crucial.

DeepSeek is pushing the boundaries of AI development by tackling the challenge of training models that think more like humans, focusing on reasoning and reinforcement learning. The company's latest models, DeepSeek R.1 and R.1.0, have shown impressive performance in reasoning tasks without relying on traditional pretraining methods, with competitive results even surpassing OpenAI's O 1 model. These models use reinforcement learning to refine reasoning, guided by rewards for accuracy and formatting, leading to the emergence of "thinking brackets" and self-correction during reasoning processes. To improve readability, DeepSeek introduced supervised fine-tuning and improved readability features. The team has also distilled massive models into smaller, more efficient versions, making them ideal for local deployment where speed and resource efficiency matter. With potential applications in enterprise AI, prompt engineering, privacy-focused AI, traditional ML tasks, and AI agents & tool use, DeepSeek's innovative approach to reinforcement learning is redefining the boundaries of AI development.

Arize AI has raised $70 million in Series C funding to accelerate its mission of building the gold standard for AI evaluation and observability. The company aims to ensure LLMs and AI agents work reliably at scale in the real world, as AI takes on high-stakes roles in finance, healthcare, and autonomous systems. Arize is developing a unified platform that combines evaluation and observability, providing a framework-independent solution for AI engineers to debug, monitor, and optimize AI systems. The company also plans to expand its partnership with Microsoft, deepen technical integrations with Google Cloud and NVIDIA's AI microservices, and hire world-class engineers to shape the future of AI observability. With this funding, Arize is doubling down on its mission to make AI work responsibly, explainably, and in ways that amplify human decision-making.

A software system that orchestrates multiple processing steps is referred to as an agent. An agent can traverse a wide solution space, handle decision-making logic, remember intermediate steps, and determine which actions to take and in what order. This enables agents to complete difficult, broad-ranging tasks that are out of the realm of possibility for traditional software. Agents shine when your application requires iterative workflows, adaptive logic, or exploring multiple pathways. For simpler use cases like basic queries, an LLM alone may be enough. However, agents offer memory and planning capabilities, tool access, and longevity and learning through iterative feedback. Ultimately, the decision to build an agent depends on the task's complexity, available resources, and the added value an agent can bring. Function calling is a key ingredient in many agent systems, where an LLM outputs structured data that maps to specific actions or APIs. This process bridges natural language with programmatic actions. Single-step choice allows the model to select the appropriate function based on user input, while structured output follows a specific format making it easier to parse and reducing errors. Scalability is also achieved by adding more functions to handle additional tasks without altering core logic. For more complex tasks, you can expand the LLM's role to include iterative reasoning and tool usage. This includes single call with function calling for straightforward tasks where the LLM needs to pick the right tool once, and agent with memory and multi-step reasoning for orchestrating complex workflows and calling different tools as new information comes in. Both approaches leverage function calling but differ in their layer of logic, memory, and iterative planning. You can either build your agent from scratch or use a framework like smolagents, LangGraph, or AutoGen, which handle common challenges such as state, routing, and skill descriptions while offering established best practices and examples. Frameworks are helpful for quick setup, having great resources, and integrating seamlessly into orchestration libraries. However, they may reduce flexibility, be opinionated in their designs, and cause lock-in concerns when switching away. For highly specialized or large-scale architectures, coding your own agent might be a better fit as it allows fine-tuning every layer and avoiding limitations imposed by a framework's design. The rest of the guide will show how to set up an example agent using smolagents, AutoGen, or LangGraph. Smolagents is great because it has pre-built common agents, seamless integration with Hugging Face tools and infrastructure, and a flexible architecture that allows for both simple function-calling agents and more intricate workflows. By following the steps outlined in this guide, you can build your own functioning AI agent using smolagents, AutoGen, or LangGraph. You'll learn how to set up an example agent, explore how quickly you can get an agent running using Hugging Face's smolagents, walk through the setup, configuration, and a simple example to highlight just how straightforward it can be. The guide will cover building an agent using smolagents, AutoGen, or LangGraph, including installing required libraries, importing essential building blocks, setting up OpenAI API key and model, creating helper functions for extracting content, defining nodes, and compiling the workflow, as well as invoking the workflow and retrieving final output. By following this guide, you'll have a basic agent that can tackle real-world tasks with minimal manual intervention. As the space evolves, keep experimenting, stay flexible, and refine your approach to deliver the best possible user experience.

The Arize release notes highlight several key enhancements, including the ability to schedule monitors to run at specific times, reduce SDK export time by exporting only desired columns, and create datasets from CSV files. The monitors have also been improved with a sleek design, added search and sorting functionality, and new monitor types such as performance and data quality monitors. Additionally, support has been added for OTEL tracing via HTTP protocol, allowing users to send traces to Arize through an OTEL tracer. The release notes also mention the addition of new content, including video tutorials, paper readings, ebooks, self-guided learning modules, and technical posts.

100X AI is a startup that's building AI agents to help engineering teams resolve incidents faster and with greater precision. They use Arize Phoenix for observability, tracing, and performance monitoring, which helps them fine-tune their AI agents and close the gap between alerts and resolution. 100X AI aims to address the knowledge problem in troubleshooting by providing an agent that can work together to form a holistic view of the system, making it easier for engineers to solve problems quickly.

Agentic RAG, a variation of Retrieval-Augmented Generation (RAG), introduces intelligent agents into the retrieval process to handle complex queries across multiple data sources. These agents can determine if external knowledge sources are needed, choose specific data sources to query, evaluate retrieved context, and decide on alternative retrieval strategies. Agentic RAG can be implemented in two ways: single agent managing all operations or multi-agent handling different aspects of retrieval. A practical implementation using LlamaIndex's ReAct agent framework combined with vector and SQL query tools demonstrates the potential of Agentic RAG. Monitoring and observability are crucial for improving system performance, and tools like Arize Phoenix can help by tracing query paths, monitoring document retrieval accuracy, and identifying improvements in retrieval strategies. Implementing Agentic RAG requires clear tool descriptions, robust testing, high-quality knowledge base documents, and a comprehensive monitoring strategy.

This novel approach to language model fine-tuning introduces a multiagent framework that leverages a team of specialized models with distinct roles, promoting diversity in reasoning and sustaining long-term performance gains. By employing a society of agents with varied responsibilities, such as generation and criticism, the system iteratively improves itself through autonomous self-improvement, achieving significant performance gains across various reasoning tasks. This method has been successfully tested on both open-source and proprietary models, demonstrating its versatility and broad applicability. The framework maintains response diversity by ensuring each agent is trained only on its own correct responses, mitigating the collapse into uniform outputs often seen in single-agent fine-tuning. However, challenges such as maintaining diversity, coordinating individual performance with system effectiveness, and optimizing computational resources remain to be addressed.