June 2025 Summaries

Multi-cloud Capacity Management (MCM) is an innovative orchestration layer designed to unify GPU resources across multiple cloud providers and regions into a single elastic pool, optimizing for high uptime, reliability, and low latency. MCM transforms traditionally siloed compute environments by treating disparate clusters and regions as a globally fungible resource pool, enabling seamless autoscaling and failover while mitigating single points of failure. Built over six months by an infrastructure team, MCM leverages Kubernetes for a global, self-healing scheduling system that adapts to real-time capacity needs, ensuring the consistent performance of AI models across different clouds. By forming partnerships with over 10 cloud providers, MCM offers virtually unlimited scalability and capacity, making it an essential tool for enterprises seeking to manage complex, high-demand workloads without the operational overhead of manual resource allocation. While MCM sets a new standard for AI infrastructure, its complexity and resource requirements suggest that similar solutions should be pursued by inference providers rather than being developed in-house by individual companies.

The Baseten Performance Client is an open-source Python library that improves throughput for high-volume embedding tasks by releasing the Global Interpreter Lock (GIL) during network-bound tasks, allowing true parallel request execution. This results in lower latencies under heavy loads, with a 12x speedup compared to the standard AsyncOpenAI client at extreme scale. The client is compatible with OpenAI and other inference providers, and its architecture utilizes multi-core CPUs to maximize throughput. It can be easily integrated into existing codebases and supports both synchronous and asynchronous usage, making it suitable for various use cases such as embedding large datasets or serving thousands of embedding queries in parallel.

The text discusses the concept of Forward Deployed Engineering (FDE), a function that sits within the engineering organization of a company and works directly with customers' engineers to accelerate time-to-value with the product. FDE is distinct from other functions such as consulting, solutions architecture, or sales engineering, as it makes regular contributions to the product roadmap and operates at the boundary between the product and customer. The text highlights that FDE is suitable for companies with substantial product and/or technology risk, deep technical stakeholders, and high leverage from customer-driven innovation. It also emphasizes the importance of recruiting, enabling, and retaining top-tier engineers who can work closely with customers and make significant contributions to the product. The author shares lessons learned from running an FDE team at Baseten, including engaging with customers early, keeping FDE within engineering, hiring builders and letting them build, prioritizing software engineering fundamentals, and contributing frequently to the product. Ultimately, the text suggests that FDE can be a transformative solution for companies with highly technical products and a startup ethos, but it's not a one-size-fits-all approach and requires careful consideration of its suitability for each company.

Baseten's MCM system is a unified control layer that provisions and scales thousands of GPUs across multiple clouds and regions. The system offers three deployment modes: Baseten Cloud, Self-hosted, and Hybrid, each sharing the same inference stack. Baseten Cloud provides fully managed, multi-cloud scale and latency optimization, while Self-hosted allows for full control over data, compute, and networking. The Hybrid model combines self-hosting with optional, elastic spillover to Baseten Cloud for dynamic routing and on-demand flex capacity. The MCM system delivers 99.99% uptime, lowest-possible latency, data-residency compliance, and freedom from vendor lock-in, making it suitable for various workloads and use cases.