November 2025 Summaries

Langfuse's November 2025 update introduced significant enhancements across various features, including Agent Observability, Model Pricing Tiers, Score Analytics, and the new MCP Server. The updates to Agent Observability enhance agent tracing with improved tool call visibility, a unified Trace Log View, and more observation types, while the introduction of Model Pricing Tiers allows for accurate cost calculations based on context-dependent pricing. Score Analytics offers comprehensive tools for analyzing evaluation scores, aiding in the validation of LLM applications, and Schema Enforcement ensures dataset consistency. The new hosted MCP Server supports prompt management, and Langfuse also introduced new integrations and numerous fixes and improvements across the platform. Additionally, Merck, a prominent pharmaceutical company, leverages Langfuse to power over 80 AI project teams globally, highlighting the platform's role in turning generative AI models into transparent and optimizable assets.

Building a custom annotation UI tailored to specific domain needs is both efficient and effective when using an existing data infrastructure like Langfuse, which provides clean APIs and TypeScript SDKs to streamline the process. Custom UIs address the unique requirements of non-technical reviewers, minimizing cognitive load and increasing productivity by eliminating unnecessary platform complexities and including useful features like keyboard shortcuts and domain-specific renderings. This approach is particularly advantageous when platform complexity hinders workflow efficiency and when reviewers require domain-specific data presentations. The process begins by identifying necessary API endpoints for queue management and score submissions, allowing for a focused development effort that avoids unnecessary complexity. Using tools like Claude and v0.dev, developers can rapidly prototype and deploy a functional UI, leveraging the existing infrastructure to handle backend operations, thereby concentrating on creating an intuitive and seamless user experience.

On November 18, 2025, Langfuse Cloud experienced a significant multi-hour outage from 11:33 am to 2:40 pm UTC due to a global incident with Cloudflare, which affected all API and UI traffic routed through its proxy. This incident exposed Langfuse's overreliance on Cloudflare as a single point of failure, causing major service disruptions and resulting in the loss of observability data and unavailability of key services like the Langfuse Prompt Management API. Despite early alerts and partial service recovery, immediate mitigation was hindered by the inaccessibility of the Cloudflare dashboard, which prevented configuration changes. As a temporary measure, DNS-based bypasses were published to allow direct access to AWS Application Load Balancers, but a full resolution was only achieved by 2:40 pm. Moving forward, Langfuse plans to diversify its infrastructure to avoid such dependencies and ensure greater resilience, including assessing alternatives for WAF and proxy functions and considering migration of DNS management and status page hosting away from Cloudflare's infrastructure. The incident highlighted the critical need for infrastructure diversification to prevent future outages, and Langfuse has committed to implementing changes to avoid similar disruptions.

Evaluating applications powered by large language models (LLMs) requires a systematic approach to ensure reliable performance, involving observability, error analysis, automated evaluation, testing, synthetic datasets, and experiments. Observability is crucial for transforming LLMs into inspectable systems by logging inputs, outputs, latencies, and metadata, which allows developers to spot patterns and measure improvements. Error analysis helps identify and categorize issues by reviewing traces and uncovering root causes, while automated evaluators provide a scalable way to monitor failure modes and integrate evaluations into CI/CD pipelines. Testing combines deterministic and probabilistic checks to prevent regressions and complements observability, while synthetic datasets enhance test coverage by generating diverse inputs. Experiments allow for the comparison of different variants to quantify progress, with interpretation linking results back to error patterns and observability data. The methodology can be tailored to the complexity of the application, whether it involves simple queries or complex multi-turn conversations and agent-based systems.

AI systems, due to their stochastic nature, require a unique approach to testing and validation, distinct from traditional deterministic software. Langfuse provides a framework for running and interpreting experiments, allowing developers to evaluate AI applications systematically. This involves defining tasks, utilizing datasets, and employing evaluators to score output quality, while managing factors like cost and latency. The guide emphasizes a structured approach akin to a CI pipeline for model quality, where experiments are executed and results are interpreted through a top-down funnel of macro metrics, baseline comparisons, and root cause analysis. Human annotation plays a critical role in refining automated evaluations, turning regression signals into structured datasets for further iterations. This systematic evaluation process is essential for developing reliable and high-quality AI applications.