February 2025 Summaries

Baseten has announced a $75M Series C funding round, co-led by IVP and Spark, to further its mission of building the world's best inference platform for mission-critical AI workloads. The company aims to address the challenges of deploying models at scale, including applied model performance research, elastically scaling infrastructure, and model management and observability. With this funding, Baseten plans to invest heavily in R&D, expand geographically, and grow its team to build a platform that enables developers to build fast, reliable, and scalable AI solutions. The company has already made significant progress, working with fast-growing AI companies and serving millions of end customers worldwide.

Multi-node inference is a technique used to serve large language models like DeepSeek-R1 by recruiting multiple high-performance GPUs to process a single model. This approach overcomes the memory constraints of individual GPU nodes, allowing for production-ready deployment on widely available H100 GPUs. However, it introduces new infrastructure and performance challenges, including ensuring consistent inter-node communication and optimizing model parallelism for efficient inference across multiple GPUs. To overcome these challenges, Baseten has developed production-ready multi-node inference solutions, enabling customers to run mission-critical workloads on scalable, cloud-agnostic infrastructure.

Baseten Chains is a framework and SDK for serving highly-performant compound AI systems in production. It enables AI builders to deploy ultra-low-latency compound AI with unique hardware and autoscaling for each step, eliminating performance bottlenecks and model orchestration headaches while keeping inference cost-efficient. The solution addresses challenges such as model orchestration, latency, reliability, and cost associated with deploying compound AI systems in production. With improved performance and developer tooling since beta, Baseten Chains is now generally available for production use, allowing users to build any custom, multi-step, or multi-model AI workflow while gaining the model performance and fluid horizontal scaling that Baseten specializes in. The platform provides a developer experience with features such as output streaming, binary IO, subclassing, Chains Watch, and a linter, making it easier for developers to deploy performant compound AI systems in production.

DeepSeek-R1, a private and secure deep learning model, is now in production in US and EU data centers, challenging the assumption that training massive closed-source models like GPT-4o and o1 is a sustainable moat. The sudden advent of open-source models with strong benchmarks compared to OpenAI’s latest offerings signals a future where every business can have its own high-performance foundation models. Companies face three options for how they’ll respond to this shift: staying on OpenAI/Anthropic, switching to DeepSeek on a shared API endpoint, or running DeepSeek-R1 on a dedicated deployment, which requires a major engineering effort but offers the only viable option due to massive security and compliance risks of shared endpoints. Baseten is uniquely positioned to help companies take advantage of DeepSeek without the typical roadblocks by providing dedicated deployments on their multi-cloud infrastructure, supporting high-performance multi-node inference, and offering enterprise-grade performance with custom optimizations on top of fast inference frameworks.

The NVIDIA GH200 Grace Hopper Superchip is a unique architecture that combines an NVIDIA Hopper GPU with an ARM CPU via NVLink-C2C, promising advantages for AI inference workloads requiring large KV cache allocations. The GH200's high-speed interconnect allows offloading parts of the KV cache to abundant CPU memory, unlocking optimizations like prefix caching and KV cache re-use. In experiments serving Llama 3.3 70B on a single 96GB GH200 GPU, the superchip outperformed an H100 GPU by 32%, with performance gains coming from access to a larger KV cache rather than just higher VRAM bandwidth or identical compute profiles. The results suggest that the GH200 Superchip is well-suited for high-throughput deployments of models that wouldn't fit on standalone GPUs with similar VRAM profiles, and its unique architecture powers the GB200 Grace Blackwell Superchip, which promises to be extremely powerful for model inference and supports multi-node NVLink for serving larger models.