As companies increasingly develop applications powered by large language models (LLMs), they face challenges in monitoring and assessing the quality of AI-generated outputs. Traceloop addresses these challenges with OpenLLMetry, an open-source project that captures real-time data such as prompts and completions from LLMs, sending it to their observability platform for analysis. This platform provides metrics to evaluate quality aspects like relevance and safety, and helps identify issues such as hallucinations or poor-quality responses. Collaborating with companies like Microsoft and Apple, Traceloop is establishing OpenTelemetry conventions for LLM observability to support the responsible adoption of complex AI systems.