December 2024 Summaries

The top 10 most loved features of 2024` were focused on making Braintrust, a platform for AI model development and analysis, simpler, faster, and more powerful. The platform introduced custom scorer, tool, and prompt functions, as well as an AI proxy and hybrid self-hosting capabilities. Structured outputs in the playground were also improved, along with monitoring improvements that included sparkline charts and flexible resizable charts. Faster experiment and log loading speeds were achieved, along with human review options and custom provider configuration. Additional features included improved logs and search functionality, attachment uploads and previews, as well as flexible visualizations for comparing performance differences. The platform's developers expressed gratitude to users who provided feedback and reported issues, which guided the development of these improvements.

The new monitor page provides comprehensive performance and usage patterns across both logs and experiments, allowing users to easily analyze key metrics such as latency, token counts, and cost. This tool is designed to help with debugging performance issues, optimizing costs, tracking prompt changes, and monitoring production stability. The monitor page offers flexible data analysis options, including grouping by metadata fields, applying filters, analyzing across different timeframes, drilling down into specific datapoints, and viewing corresponding traces. It is now available in all projects for users to access and provide further metric suggestions.

Every week, it seems like another AI provider releases a state-of-the-art model. These announcements come with impressive benchmarks, but those benchmarks rarely reflect real-world use cases. So, how do you know if the new model is worth deploying in your app? To gauge if a particular model will improve your application, it’s first worth understanding how well your app is currently performing by setting up a baseline using evaluations that consider the accuracy or quality of the LLM outputs. The best way to evaluate a new AI model is by testing it against the actual data your app handles in production, pulling real logs from your app and organizing them into a dataset. If the results show that the new model outperforms your current one, update it in production with just a one-line code change. After shipping the new model in production, you can keep tabs on its performance on the Monitor page, selecting Group by model to focus on your model change and tightening the timeline to when you made the changes. By testing a new AI model with your actual data and swapping models easily, you’ll know for sure if it’s better for your app.