November 2025 Summaries

The blog post discusses the launch and technical advancements of the Model API for Kimi K2 Thinking, an open-source model that achieves state-of-the-art performance with a latency of 300 milliseconds and over 140 tokens per second, as measured by Artificial Analysis. Utilizing the Baseten Inference Stack, the model runs on a single 8xB200 node in NVFP4, incorporating Tensor Parallelism, Expert Parallelism, and KV-aware routing to enhance performance and cache reuse rates. Kimi K2 Thinking, comparable to leading models like GPT-5 and Claude Sonnet 4.5, stands out for its speed and cost-effectiveness while bridging the intelligence gap between open-source and closed models. The use of NVIDIA Blackwell GPUs, despite challenges in converting data formats from INT4 to NVFP4, is highlighted as crucial in achieving high throughput and low latency. The deployment includes advanced parallelism strategies and KV cache re-use to optimize processing for complex AI tasks, such as code generation and long-context queries. The post also notes ongoing performance improvements and future enhancements to the model's functionality and quality.

The rise of AI agents has led to a proliferation of open-source models that support tool calling, a method enabling large language models (LLMs) to interact with external applications for more dynamic and efficient performance. This development has highlighted the varying quality of tool calling among inference providers, with benchmarks becoming crucial in evaluating their success. Tool calling enhances model efficiency by allowing external task handling, thereby extending the model's relevance without frequent retraining. It involves single-turn and multi-turn interactions, with the latter introducing complexities that can affect output quality. Inference providers play a pivotal role in pre-processing, model execution, and post-processing to ensure successful tool calling, with techniques such as structured outputs, quantization, and parsing being vital. Baseten emerges as a notable platform in this space, emphasizing reliability and performance in tool calling through comprehensive pre-processing, model execution, and post-processing strategies, as demonstrated by their success in recent benchmarks.