September 2024 Summaries

The Braintrust Playground now offers the ability to create custom scoring functions, allowing developers and non-technical builders to move faster, run more experiments, and build better AI products. This feature enables users to access custom scorers via the UI and API, unlocking new capabilities such as running sophisticated comparisons across multiple prompts and scoring functions, defined as LLM-as-a-judge, TypeScript, Python, or HTTP endpoints. The custom scorer functionality can be created using a combination of heuristics (best expressed as code) and LLM-as-a-judge (best expressed as a prompt), and can also utilize existing evaluators in autoevals as a starting point. Users can upload task and scorer functions to Braintrust from the command line, access custom scorers through the API, and run server-side online evaluations asynchronously on specific logs. This new feature unlocks several workflows, including iterating on prompts, monitoring performance over time, finding anomalous cases, and adding logs to a dataset for additional evaluations.

The latest feature in Braintrust is the implementation of Copilot autocomplete in its UI. This feature allows users to leverage their existing API keys with models like gpt-4o-2024-08-06, Claude 3.5 Sonnet, LLaMa 3.1, Gemini, Mistral and others. The autocomplete is enabled for everyone who has configured at least one AI provider key, and it uses the user's selected model to provide intelligent completions based on context from other data fields. Braintrust has captured completion data from its staging account and run initial evaluations which showed that the default model performs almost as well as some of the more expensive options, with room for improvement through additional data and customer feedback. The new feature is available for everyone who already has a Braintrust account or can sign up for free to try it out, and users are encouraged to provide feedback to help improve the feature.