May 2024 Summaries

As a forward-deployed engineer (FDE) at Baseten, I've learned to tackle complex technical challenges daily, partner with customers to maximize the value of their AI projects, and deliver rock-solid solutions that exceed customer expectations. With 75% software engineering, 15% technical consulting, and 10% customer relationship management, my role requires diving deep into complex technical challenges, pushing the boundaries of ML engineering, and delivering high-quality solutions. I've honed skills in optimizing models for performance, leveraging advanced tools and techniques to boost performance, and ensuring robust production AI services. As an FDE, I measure impact by shipping successful solutions that deliver new value to customers and drive revenue growth, while also being mindful of the importance of context-switching and prioritizing customer satisfaction.

The concept of control plane and workload plane in model serving infrastructure is a powerful abstraction that enables building worldwide multi-cloud AI model serving infrastructure. The control plane, a single Kubernetes cluster, serves as the backend for user interface and model management API endpoints, while also handling tasks such as building model serving images and balancing load across workload planes. In contrast, the workload planes are collections of GPU resources for running model inference, which can be set up in arbitrary cloud environments and regions. The separation of concerns between control and workload planes is motivated by customer and operational challenges, including regional preference, GPU availability, self-hosted model inference, scaling with customer demand, and reducing compute and maintenance overhead. Each workload plane has its unique capabilities and limitations, affecting the overall system's performance and security. By separating data from control or workers from a centralized decision maker, the control plane is able to orchestrate workload planes across regions, cloud providers, and cloud accounts, while each workload plane adjusts to the specific environment it's running in.

Comparing tokens per second across LLMs is crucial to accurately evaluate the performance of Large Language Models (LLMs) during inference. The efficiency of tokenizers used by different models varies widely, with some being more efficient than others in processing human-readable input text and generating output as tokens. When comparing performance across two different LLMs, it's essential to adjust Token Per Second (TPS) metrics based on the models' tokenizers to ensure accurate comparisons. Different LLMs have varying levels of efficiency in their tokenizers, with some being better suited for specific use cases such as code or prose. To accurately calculate changes in latency, throughput, and cost when switching between open source models, it's necessary to adjust TPS calculations to reflect real-world use and account for the relative value of each token generated. By doing so, developers can set accurate performance targets and optimize their LLMs for better performance.

In April 2024, Baseten released several best-in-class large language models (LLMs) in different sizes, ranging from 3.8 billion to 141 billion parameters, offering flexibility for trade-offs between cost and output quality. These models can be deployed with optimized techniques such as TensorRT-LLM implementations, FP8 quantization, and continuous batching for improved inference efficiency. Additionally, Baseten introduced streaming endpoints for its LLMs, enabling real-time text-to-speech synthesis capabilities. The company also emphasized the importance of CI/CD pipelines for AI models, providing a model management API to build customized tooling for deployment, and introducing a new feature that allows for more reliable deployment status tracking. These updates expand the possibilities for building with AI and improve the stability of production deployments.