November 2023 Summaries

The Braintrust AI Proxy is a free, open-source tool that allows developers to access various machine learning models behind the OpenAI protocol with built-in caching and API key management. The proxy's source code has been made available on GitHub under the MIT license, enabling production deployment. Users can continue to access the proxy via its hosted version or deploy it themselves using instructions provided in the repository. Performance benchmarks have shown that Cloudflare and Vercel are consistently fast options, while AWS Lambda experiences slower performance. The proxy includes features such as caching, API key management, load balancing, and integration with Azure OpenAI endpoints. An exciting roadmap for the proxy is planned, including advanced resiliency features, support for additional models/providers, and deeper integrations into Braintrust.

The AI proxy is a new feature announced by Braintrust, an institution that fosters an open ecosystem for the AI industry. The proxy aims to address challenges in LLM development such as interoperability and API key management by embracing OpenAI's interface as the lingua franca for LLMs. It adds caching, logging, and API key management behind the scenes, supporting popular open source models like LLaMa 2 and Mistral via Perplexity. The proxy enables users to build robust, low-latency systems that work across a thriving ecosystem of model providers without changing code, promoting interoperability and defaulting to openness. It is available for all to use as a beta, for free, with features such as caching, logging, and API key management, and is expandable to include more providers and features, encouraging collaboration and feedback from users.

Most companies are adopting AI features and putting them into production, with 66.2% having at least one internal use case live and 43.1% having at least one external use case live. However, building and productionizing AI apps is challenging due to issues such as model output accuracy, hallucinations, and prompt engineering. Companies are seeking ways to evaluate their AI apps, including how to integrate evaluations with Continuous Integration and Continuous Deployment (CICD) systems and how to score AI outputs using tools like GPT-4. Many companies are either manually checking their AI's work or not checking it at all, highlighting the need for efficient evaluation tools. Braintrust offers a platform that enables users to add evaluations to their AI apps in under 10 minutes, providing a convenient solution for companies looking to streamline their internal tool development.

We shipped user experience improvements and fixes to make it easier for developers to build reliable AI apps. We added function calling to the prompt playground, which allows users to experiment with OpenAI functions. The release notes highlight several other improvements, including persistent resized widths in the experiment column, fixing libraries including Autoevals to work with OpenAI's new libraries, and adding support for function calling and tools in the prompt playground. Braintrust is an enterprise-grade stack for building AI products, aiming to reduce uncertainty and tedium in incorporating AI into businesses. The platform offers various features such as evaluations, prompt playground, and data management, and invites developers to sign up now or check out their pricing page.

Building reliable AI apps is challenging due to the lack of best practices for testing and evaluation. The traditional software development approach, such as setting up CI/CD and writing tests, does not directly apply to AI app development. A good evaluation system is crucial in this process, but it's often unclear how to create one. This journey from prototype to production involves manual testing, getting friends to test, fixing bugs, adding features, and finally using an evaluation script to automate the testing and validation process. Braintrust provides libraries and a web UI to make evaluating AI apps easy, saving time and improving developer iteration speed. With this tool, teams can set up their evaluation workflow in under 10 minutes and focus on building fun parts of AI apps.

We've been working on an exciting new feature at Braintrust and have shipped some user experience improvements and fixes this week. We added open source models like Mistral, Codellama, Llama2, etc. to the playground for comparison with OpenAI's new models using Perplexity's API. These new additions improve the speed of our platform. Additionally, we've made improvements such as selectors for diffing and comparison modes on experiment view and added support for new OpenAI models in the playground. Braintrust is an enterprise-grade stack for building AI products, offering solutions from evaluations to data management, aiming to reduce uncertainty and tedium in incorporating AI into businesses.