March 2025 Summaries

We built Baseten Embedding Inference (BEI), an optimized inference runtime leveraging TensorRT-LLM to significantly boost throughput and minimize latency for embedding, reranker, and classification models. BEI outperforms the competition by a large margin, offering double the throughput of previous industry standards in batch inference and improved latency for real-time queries. The core engine for BEI is TensorRT-LLM, which offers exceptional performance and consistent throughput without the risk of OOM errors. BEI has four main components: the model server, tokenizer, batch manager, and TensorRT-LLM inference engine. The runtime benefits from optimized inference engines, such as XQA kernel and layer fusing, as well as quantization to FP8, which provides a 50% or more gain in throughput while retaining >99% cosine similarity to outputs from non-quantized models. BEI supports traffic-based autoscaling, deployment on multiple clouds and regions, and reduced communication overhead between models.

Baseten Embeddings Inference (BEI) is the fastest embeddings solution available for high-throughput and low-latency production workloads. It offers over 2x higher throughput and 10% lower latency compared to previous industry standards, making it suitable for rapid responses in applications such as search and retrieval, agents, and recommender systems. BEI is designed to provide optimized inference performance out of the box for embedding, reranker, and classification models, with a focus on low memory footprint and scalability. It can be used with open-source, custom, or fine-tuned models, and works well in compound AI systems, making it an ideal solution for companies building products that leverage embeddings in production.