January 2025 Summaries

Fintool, an AI equity research assistant, uses the platform Braintrust to generate millions of financial insights daily, serving leading institutional investors and companies like PricewaterhouseCoopers. To address the challenge of managing large volumes of unstructured data, Fintool developed a Twitter-like interface called Fintool Feed, which summarizes key sections of documents based on user prompts. The team faced challenges such as managing massive amounts of data, ensuring accuracy and reliability, and implementing real-time monitoring. To overcome these challenges, Fintool used Braintrust to develop a continuous evaluation workflow that includes defining quality standards and format rules, curating golden datasets, automating evaluations with LLM-as-a-judge, and adding human oversight. This workflow has helped Fintool improve accuracy, consistency, efficiency, and scalability, delivering reliable financial insights at scale while maintaining trust in high-stakes financial contexts.

The Loom team developed a robust method for evaluating the quality of their auto-generated video titles using generative AI. They started by identifying key traits of great video titles and checking for common measures of quality across various use cases. Next, they implemented objective measures with code-based scorers to automate these quality checks, removing variability from LLM responses. The team then created initial scorers and iterated on them by feeding in test examples, refining as needed. Through a cycle of defining criteria, implementing scoring functions, evaluating results, and refining, the Loom team established a repeatable system for shipping features faster and more confidently using Braintrust evals.

This blog post provides a comprehensive guide on evaluating the quality and accuracy of agentic systems, which are complex systems that can perform tasks autonomously. The authors highlight the importance of running evaluations to detect and debug issues before they impact users, and provide practical strategies for choosing evaluation metrics, building block: the augmented LLM, prompt chaining, routing, parallelization, orchestrator-workers, evaluator-optimizer, fully autonomous agents, best practices, and next steps. The post covers various types of agentic systems, including simple augmented large language models (LLMs), fully autonomous agents, and more complex systems that combine multiple components. It also discusses the challenges of evaluating these systems, such as determining the right set of scorers, handling subjective or contextual feedback, and incorporating domain-specific knowledge. The post concludes by emphasizing the importance of refining or replacing scorers over time to learn more about the real-world behaviors of agentic systems at scale.

When using Braintrust as part of an LLM development workflow, users have the flexibility and control over their data by deploying it in a hybrid mode. This approach allows for the hosting of sensitive data in the user's own environment while keeping the UI and metadata hosted by Braintrust. The architecture consists of two main components: the data plane and the control plane, where the data plane handles the actual data and is hosted in the user's environment, while the control plane serves the UI and metadata and is hosted by Braintrust. This hybrid model provides security and compliance with the ability to update features automatically, configure rate limits, custom URLs, and domain proxies to meet IT environment and security policies. Users can self-host on their preferred infrastructure, including AWS, using pre-provided configuration files or templates, and receive support for advanced deployment scenarios.