April 2025 Summaries

In the AI inference landscape, NVIDIA B200 GPUs have emerged as a game-changer for accelerating high-traffic endpoints with 5x higher throughput, more than 50% lower cost per token, and up to 38% lower latency for serving large LLMs like DeepSeek-R1. The new architecture offers better raw specs, faster inference frameworks like TensorRT-LLM, SGLang, and vLLM, and even improved FP4 quantization for efficient and accurate inference. To unlock these benefits, developers need to consider model performance optimization, distributed GPU infrastructure, model management tooling, and AI engineering expertise. With Baseten's support, users can get started with B200 GPUs today and accelerate their inference workloads for huge volumes of traffic.

You can now use Chroma, the open-source AI application database, with Baseten's inference platform to create AI-native apps. Chroma is unique among vector databases because it is open-source and offers a natural choice for developers building with open models who want control over their entire AI infrastructure stack. Baseten offers dedicated deployments of every open-source, fine-tuned, and custom embedding model on autoscaling infrastructure. Recently, Baseten announced Baseten Embedding Inference (BEI), the world's fastest runtime for embedding models, which provides twice the throughput of the previous leading solutions for modern LLM-based embedding models. BEI is useful with Chroma in two ways: it offers substantial speed and cost savings when filling the Chroma vector database with an initial corpus of data, and low-latency, real-time embedding inference when passing user queries to the Chroma database. You can use BEI-optimized embedding models deployed on Baseten with Chroma via our official integration. To use Chroma with Baseten, you can call an embedding model running on Baseten using the Chroma Python SDK in less than five minutes, by deploying an embedding model on Baseten, installing the Chroma Python package, and generating embeddings using Baseten and Chroma. This makes building AI-native apps simpler and faster with Chroma's open-source flexibility paired with Baseten's high-performance embedding inference (BEI).

The article provides a curated list of the best open-source embedding models for various applications such as agents, RAG pipelines, search, and recommendations. The top recommended models include BAAI bge-en-icl, Mixedbread Embed Large V1, Nomic Embed Code, BAAI bge-reranker-v2-m3, and Tulu 3 8B Reward. These models offer a balance of accuracy, embedding speed, and cost, making them suitable for different use cases. The article also addresses common questions such as what can be built with embedding models, how to optimize open-source models for latency and throughput, and the importance of choosing an embedding model that performs well on your specific data.