August 2025 Summaries

HTTP, WebSockets, and gRPC each serve distinct roles in client-server communication, offering different strengths depending on application needs. HTTP is renowned for its simplicity and reliability in request-response interactions, making it compatible with existing internet infrastructure such as proxies and firewalls. This stateless protocol is ideal for scenarios requiring discrete interactions and caching capabilities. WebSockets, on the other hand, establish a persistent, bidirectional connection, making them suitable for real-time applications like interactive AI, though they require more complex error handling due to the potential loss of shared state if connections drop. gRPC, a high-performance framework developed by Google, leverages Protocol Buffers for strong type safety and schema validation, excelling in service-to-service communications within distributed systems, with its foundation on HTTP/2 enabling bidirectional streaming and enhanced efficiency. Each protocol presents trade-offs, and the choice largely depends on the specific requirements of the application, such as the need for real-time interaction, type safety, or compatibility with existing infrastructure.

Dannie Herzberg has been appointed as the new President of Baseten, bringing a wealth of experience from her roles at HubSpot, Slack, and Sequoia to lead the company's go-to-market and operational strategies. Her expertise in scaling companies and her customer-centric approach align with Baseten's commitment to delivering the fastest, most reliable inference and exceptional developer experiences. At HubSpot, she contributed to shaping the go-to-market strategy, and at Slack, she played a key role in scaling the company from $100 million to over $1 billion in revenue before its IPO. Her time at Sequoia provided her with insights into building enduring companies, making her an ideal fit for Baseten's mission of leveraging AI as a significant economic opportunity. Her leadership is expected to enhance Baseten's ability to respond with speed and discipline to the rapidly evolving AI landscape, reinforcing the company's dedication to customer success.

OpenAI has released two new open-weight models, gpt-oss-120b and gpt-oss-20b, their first since GPT-2 in 2019, with the larger model rivaling OpenAI’s proprietary o4-mini on reasoning benchmarks and the smaller performing on par with o3-mini. These models can be fine-tuned for specific use cases, although fine-tuning the larger model is complex due to compute and model parallelism demands. To address these challenges, Baseten and Axolotl have partnered to provide a streamlined recipe for fine-tuning the gpt-oss-120b. Axolotl offers an open-source fine-tuning runtime that supports various techniques and is optimized for distributed parallel training, while Baseten simplifies large-scale model training by providing access to powerful GPUs, seamless scaling, and features like dataset caching and checkpointing. The fine-tuning process involves defining configurations for both Baseten and Axolotl, using a Python configuration file and a YAML file, respectively, to ensure smooth execution and performance monitoring, with tools like Truss for launching and managing training jobs.

Cline is an open-source AI coding assistant that enhances development workflows by integrating powerful language models directly into popular integrated development environments (IDEs). By partnering with Baseten's model-serving infrastructure, Cline offers users access to advanced models such as DeepSeek R1, Kimi K2, and Qwen3 Coder, optimized for various coding tasks. The integration facilitates building complex applications, debugging, and exploring new technologies, providing tools for modern software development. To use Cline, developers must install it into their IDE and configure Baseten with an API key for model access. Cline offers "Plan" and "Act" modes, allowing users to either architect solutions or implement them directly, as demonstrated by generating a browser-based Flappy Bird game. The choice of model depends on the coding scenario, with Qwen3 Coder suited for rapid prototyping, Kimi K2 for complex integrations, and DeepSeek R1 for in-depth analysis and problem-solving. Users are encouraged to experiment with the full range of models available through a single Baseten API key to enhance their development processes.

Baseten has appointed Joey Zwicker as the Head of Forward Deployed Engineering (FDE), with the aim of enhancing their AI product offerings by leveraging Joey's extensive industry experience. Joey co-founded Pachyderm, a company focused on automating data pipelines, which was later acquired by HPE, where he managed AI software products. At Baseten, Joey will lead the FDE team, emphasizing hands-on customer interaction and accountability throughout the production lifecycle to ensure reliable and cost-efficient AI solutions. His strategy involves hiring and training engineers who are adept at solving complex problems and fostering trust with both startups and large enterprises. Joey appreciates Baseten's collaborative and transparent culture and is eager to contribute to its growth by integrating operational efficiency with customer-focused product development. Baseten is actively hiring FDEs to expand their team and continue building innovative solutions.

Canopy Labs has released Orpheus, an open-source text-to-speech model that leverages WebSockets for real-time applications, such as voice-enabled customer support and language translation. Unlike traditional HTTP requests, WebSockets maintain a persistent, bidirectional connection that facilitates seamless communication between server and client, allowing for continuous data exchange without the need for repeated connection setups. Baseten offers an implementation of Orpheus that supports streaming via WebSockets, making it possible to simultaneously send text data and receive audio data in real-time. The process involves deploying the Orpheus model on Baseten, generating an API key, and using specific Python packages to establish a WebSocket connection. This setup enables the streaming of text-to-speech data, which can be tested with sample text to produce audio output through a computer speaker. The guide emphasizes the importance of appropriate autoscaling settings in production environments to accommodate growing numbers of concurrent users.

Achieving state-of-the-art (SOTA) latency and throughput for the GPT OSS 120B model on NVIDIA GPUs involves a complex process of performance optimization, including experimentation, bug fixing, and benchmarking. The Baseten Inference Stack significantly contributes to this endeavor by allowing rapid performance improvements through its flexible architecture and the expertise of its model performance engineering team. Upon the model's release, engineers work in parallel using different inference frameworks such as TensorRT-LLM, vLLM, and SGLang, ensuring compatibility with Hopper and Blackwell GPU architectures. The team addresses compatibility bugs and optimizes model configurations, choosing Tensor Parallelism for better latency and leveraging TensorRT-LLM MoE Backend for enhanced performance. These efforts lead to significant improvements, including adding 100 tokens per second while maintaining 100% uptime, and highlight the importance of inference optimization for immediate improvements in latency and throughput. The team continues to explore new methods like speculative decoding to further enhance model performance, with a focus on providing efficient solutions for developers looking to optimize their models.

Kimi K2, developed by Moonshot Labs, is a groundbreaking 1 trillion parameter AI model optimized for agentic tasks such as building agents, coding assistants, and multi-step reasoning systems. It achieves this through three main innovations: a mixture-of-experts architecture with 384 specialized experts and reduced attention heads for improved focus, a novel post-training method that generates synthetic agentic data via simulated tool interactions rather than relying solely on human data, and the MuonClip optimizer, which stabilizes training by clipping attention logits to prevent loss spikes. This model, an iteration of the DeepSeek architecture, is particularly effective in coding and agentic tasks due to its deeper expert specialization. The Kimi K2 model is made accessible through Baseten's infrastructure, and it leverages scalable agentic data synthesis, reminiscent of DeepMind's AlphaGo, to surpass traditional pretraining limitations. The MuonClip optimizer represents a significant breakthrough by eliminating training loss spikes, potentially reducing computational costs in the industry. Kimi K2's deployment is facilitated by advanced optimizations like Tensor parallelism and KV-Cache optimization, making it an attractive choice for developers interested in exploring advanced AI applications.

Whisper, an open-source transcription model by OpenAI, is optimized for real-time speech transcription through WebSockets, particularly effective for applications requiring seamless, continuous data exchange such as AI transcription and live communications. Unlike traditional HTTP request-based methods, WebSockets maintain a persistent bidirectional connection, reducing latency and connection overhead, thus enhancing the performance of streaming applications. The article details an implementation example on Baseten using Whisper V3, providing a step-by-step guide to deploy the model, configure WebSockets, and stream audio from a microphone for real-time transcription. By leveraging Python's asyncio for asynchronous operations, users can simultaneously send audio data and receive transcribed text in real time. The approach allows for scalability in production, ensuring the system can handle multiple concurrent users by setting appropriate autoscaling settings. This method underscores the utility of WebSockets in creating efficient, production-grade speech-to-text transcription systems.

Baseten has introduced Baseten Embeddings Inference (BEI) for Blackwell GPUs, which allows users to leverage the latest open-source embedding models, such as Qwen3 Embedding, and NVIDIA GPUs with optimal performance. The Qwen3 Embedding 8B model, known for its multilingual capabilities and reasoning skills, currently ranks first on the Massive Text Embedding Leaderboard with a mean task score of 70.58%. Benchmarks show that BEI on B200s provides significant performance advantages, processing 1.5 times more tokens per second than the next best solution, and offering 3.3 to 8.4 times higher throughput compared to other systems in both high and low query-throughput tests. For those interested in deploying the Qwen3 Embedding 8B or optimizing AI workloads, Baseten offers a Model Library and additional resources, including a technical deep dive and documentation.