The core KPIs of LLM performance (and how to track them)

The text discusses key performance indicators (KPIs) for evaluating the performance of large language models (LLMs) and provides insights into monitoring these metrics effectively. The author shares their experience of building an MCP server for Toronto’s Open Data portal and encountering issues with API payloads, which underscores the importance of observability. Good KPIs should provide directional signals tied to product outcomes and focus on reliability, cost efficiency, and user experience. The text highlights ten core metrics, such as agent traffic, LLM generations, tool calls, token usage, and end-to-end latency, which are crucial for understanding model performance and identifying potential failures. It emphasizes the use of observability tools like Sentry to track these metrics and suggests setting up dashboards and alerts to monitor reliability, cost efficiency, and user experience. The author advises focusing on critical metrics and maintaining operational telemetry to meet privacy needs while ensuring effective monitoring of AI agents.