September 2025 Summaries

Langfuse's September 2025 update introduced several new features and enhancements, notably the launch of their first AI feature, a new high-level SDK for experiment running, and the graduation of the TypeScript SDK v4 to general availability. The update also included structured output support for prompt experiments and a significant price reduction for the Core plan, now available at $29 per month. The Experiment Runner SDK simplifies running experiments on datasets with built-in automatic tracing and concurrent execution, while natural language filtering allows users to create filters using plain English. Additionally, Langfuse introduced a self-serve Enterprise Cloud plan for teams requiring advanced security and scalability. Khan Academy has utilized Langfuse's API to enhance its Khanmigo AI, demonstrating successful adoption across various teams for improved iteration and visibility.

In the context of AI development, setting up automated evaluations is crucial for efficiently assessing the impact of modifications to large language model (LLM) applications, as demonstrated using Langfuse. This guide emphasizes the importance of distinguishing between prompt-related errors and model limitations, advocating for automated evaluators to address the latter. It provides a framework for creating scalable evaluators, such as the LLM-as-a-Judge, which can be configured in the Langfuse UI or through custom code. The process involves drafting precise prompts, validating evaluators against human judgment using metrics like True Positive Rate (TPR) and True Negative Rate (TNR), and integrating these evaluations into a CI/CD pipeline. By consistently scoring application performance and monitoring failure modes, developers can iterate faster while maintaining high quality, ultimately improving the application’s reliability and effectiveness.