How to Trace LLM Calls in Production

Tracing large language model (LLM) calls in production involves recording detailed data about each model-powered request to understand what occurred during the process, such as the prompt, model parameters, response, latency, token usage, and any tool calls or errors. This comprehensive tracing enables teams to identify exact causes of issues, such as which prompt version or context chunk led to a user's bad answer, by providing a timeline of the entire workflow, rather than merely logging the final response. Effective tracing should encompass metadata, prompt versions, model configurations, retrieval contexts, tool calls, and output processing, while ensuring sensitive data protection and maintaining a searchable and safe trace structure. Additionally, integrating evaluations into traces helps assess the quality of outputs, and establishing production alerts for key metrics such as error rates and latency can enhance system reliability. The use of structured trace schemas allows teams to compare workflows effectively and address production issues with confidence.