Building reliable dashboard agents with Datadog LLM Observability

Datadog's Graphing AI team has integrated LLM Observability into their widget and dashboard generation agents to enhance the reliability of AI-driven systems by providing real-time visibility into their behavior. This integration allows the team to trace agent interactions, debug complex chains, and evaluate semantic and functional accuracy across different model versions. By utilizing Datadog's Experiments feature, the team automates evaluation and structured testing at scale, enabling them to identify issues swiftly and maintain high performance. The agents, which convert natural language prompts into Datadog visualizations, now benefit from improved debugging tools, allowing engineers to pinpoint failures, such as HTTP 401 responses from service calls, and address them efficiently. This system enables a reproducible offline evaluation pipeline that measures accuracy through both deterministic checks and LLM-as-a-judge assessments. Future plans include expanding the framework to evaluate models from other providers like Anthropic Claude and correlating online evaluations with Real User Monitoring data to further refine user experience metrics.