November 2025 Summaries

The text provides a guide on setting up a container image for running a server environment optimized for vLLM, which can be installed using pip due to the availability of CUDA drivers provided by Modal. The setup involves using the NVIDIA CUDA 12.8.0 development image based on Ubuntu 22.04, with Python 3.12, to benefit from optimized kernels. It details the installation of necessary packages such as vLLM, huggingface_hub with HF transfer enabled, flashinfer, and PyTorch via an extra Python package index. This configuration aims to enhance performance through optimized CUDA kernel use and facilitate faster model transfers by enabling specific environment variables.

Reducto, a company specializing in transforming unstructured documents into structured data for enterprises, successfully improved its document processing latency by 3x by adopting Modal's infrastructure. Originally relying on manually provisioned EC2 instances and later Kubernetes, Reducto faced challenges with scaling and latency due to variable workloads and high traffic spikes. Modal provided the necessary flexibility, GPU availability, and development experience to overcome these issues, allowing Reducto to independently scale models, customize scaling for individual customers, and reduce cold boot times through GPU memory snapshotting. A significant load test demonstrated Reducto's capability to scale its ingestion pipeline to over 1,000 GPUs, reinforcing its ability to handle demanding workloads. This transition also resulted in improved operational efficiency for Reducto's engineers, reducing the complexity and overhead associated with infrastructure management and enabling them to focus more on developing new AI models. Looking ahead, Reducto plans to expand its use of Modal for deploying new AI models and enhancing its document intelligence pipelines.

Host overhead, an inefficiency in AI inference workloads, occurs when the CPU delays the GPU, causing low GPU kernel utilization and increased inference costs. This issue arises when the GPU is idly waiting for the CPU's instructions, akin to a ship waiting for its navigator's directions. To mitigate host overhead, software engineers can optimize code by identifying idle periods in CUDA streams, constructing tensors directly on the GPU, and using tools like the PyTorch Profiler to trace inefficiencies. Strategies such as kernel fusion and CUDA Graphs can reduce the number of kernel launches, thereby decreasing overhead and improving performance. These optimizations are crucial in meeting the growing demand for faster AI inference in production systems. Modal is actively contributing to open-source inference engines to advance the efficiency of AI workloads, emphasizing the importance of every microsecond in performance-sensitive environments.

Decagon, in collaboration with Modal, has developed a cutting-edge real-time voice AI platform that delivers high-quality customer interactions by overcoming significant technical challenges. The launch of Decagon Voice involved creating compact and efficient AI models through advanced supervised fine-tuning and reinforcement learning, achieving state-of-the-art accuracy and response quality for voice interactions. To meet the demanding requirements of sub-second latency and seamless responsiveness, Modal's team provided deep-engine level optimizations, including custom draft models and re-engineered inference pipelines. These efforts resulted in a substantial reduction in latency and improvements in intent recognition, setting a new standard in voice AI technology. The partnership also focused on enhancing model generalization through diverse datasets and sophisticated training methodologies. Additionally, Modal's training solutions enabled frictionless model development, allowing Decagon to efficiently iterate and deploy models across various customer conversations. The collaboration underscores the complexity of building scalable, real-time AI systems and highlights the ongoing advancements in conversational intelligence.

In a detailed exploration of building a low-latency voice AI chatbot, the article outlines the use of Modal, open-source frameworks like Pipecat, and open models to achieve near real-time conversational capabilities. The chatbot architecture relies on a sequence of AI models for speech-to-text, language processing, and text-to-speech tasks, coordinated by Pipecat's voice AI framework, which supports modularity and stateful conversation management. The integration with Modal's infrastructure allows for efficient autoscaling and resource management, optimizing the use of CPUs and GPUs to keep costs low while maintaining performance. The system achieves voice-to-voice latencies around one second by leveraging Pipecat's SmallWebRTCTransport for peer-to-peer connections, Modal Tunnels for reduced network latency, and a careful selection of AI models like Parakeet for STT, Qwen3 for LLM, and Kokoro for TTS. The article also addresses strategies for minimizing latency through geographic proximity of services and discusses the challenges and solutions for maintaining performance across distributed components. Additionally, it highlights the integration of technologies for speaker diarization and analysis to fine-tune latency measurements, with practical implementation details available in a GitHub repository.