October 2026 Summaries

17 posts from Modal

Filter
Month: Year:
Post Summaries Back to Blog
Modal and Sakana AI collaborated to deploy Sakana Namazu, an approximately one-trillion-parameter mixture-of-experts model designed for Japanese business contexts, web search, and code execution, using dynamically scalable GPU infrastructure rather than reserved capacity. To reduce memory demands while preserving Japanese-language performance, they applied NVFP4 quantization selectively to routed experts while retaining BF16 precision for attention, routing, and shared experts, calibrating the model with representative Japanese-style conversations. They also developed custom speculative decoding draft models trained on more than 30 billion raw production-style token IDs, later refining the approach with data from roughly one million real requests; Modal reports that this increased token acceptance, reduced decoding latency, and improved p99 latency. Additional serving optimizations addressed CPU–GPU synchronization, expert-matrix kernels, cached setup work, multi-GPU acceptance consistency, and grammar-constrained structured outputs. After launch, a KV-cache-aware router reportedly reduced queueing and time-to-first-token spikes by directing requests toward replicas likely to contain cached prompt prefixes, while elastic GPU scaling helped accommodate uncertain launch demand.
Oct 23, 2026 1,280 words in the original blog post.
Modal has released Spindle, an open-source, Tinker-compatible post-training server that runs in a user’s Modal workspace, allowing teams to write Python training loops while retaining control over models, configurations, and GPU infrastructure. Positioned between Modal’s higher-level Dojo training library and its raw cluster tools, Spindle supports distributed full-parameter and multi-LoRA training for open models, using autoscaling, durable checkpoints, disaggregated training and inference resources, and multi-client scheduling. The project aims to reduce costs relative to token-priced closed APIs by charging for GPU time and increasing utilization through shared trainers, particularly for reinforcement-learning workloads that otherwise leave training hardware idle during rollout generation. Modal reports reward parity with Tinker on a replicated search-agent task and cites price reductions ranging from roughly 20% to 80% across math, search, and software-engineering workloads, depending on hardware and concurrency. Spindle remains in beta, is customizable through configuration changes such as expanded context lengths, and invites community contributions for model support and performance improvements.
Oct 22, 2026 1,399 words in the original blog post.
Modal Dojo is a newly available library for production-grade LLM post-training that aims to simplify reinforcement learning workflows by combining scalable infrastructure, tuned training recipes, open-source foundations, and built-in observability. It is designed for organizations with deployed agent systems, evaluation data, and production traces, enabling them to train models on proprietary datasets to improve or replace existing agents through a minimal API. Users select a base model, provide data from sources such as Hugging Face, Harbor environments, or generated production tasks, and configure a recipe governing hardware, parallelism, rollouts, environments, and other training parameters. The platform supports customization of underlying frameworks and recipes while supplying optimized defaults, dashboards for tracking rewards, losses, rollouts, timing, and logs, and CLI tools intended to help agents run and diagnose experiments. Modal cites customers including Axiom Bio and Afterquery as examples of teams using Dojo for large-scale, long-context reinforcement learning and automated research on open models.
Oct 21, 2026 1,159 words in the original blog post.
Modal’s Sticky Sessions introduce session-aware serverless compute for stateful, long-lived workloads that need every request and reconnection to reach the same container, such as speech and voice agents, multiplayer games, collaborative rooms, and stateful web interfaces. Built on Modal Servers, the feature is enabled with a `@modal.sessioned()` decorator and creates signed session tokens that authorize participants while embedding routing metadata, allowing proxies to route requests without maintaining centralized session mappings. Unlike best-effort affinity or single-container workarounds, Sticky Sessions provide fixed container routing, host-managed session lifecycles, and autoscaling based on active sessions rather than request volume. Modal’s infrastructure uses a stateless proxy, periodic capacity probes, and session queues to distribute sessions across containers and scale capacity when concurrency is exhausted. The company highlights applications including WebRTC voice bots, shared chat rooms, and its own Curvytron multiplayer game deployment, where active rooms automatically determine compute scaling.
Oct 20, 2026 1,148 words in the original blog post.
Legora rebuilt its evaluation system as it transitioned from a document-focused AI assistant to an autonomous legal agent capable of conducting hours-long work across thousands of documents. To achieve both scale and environment fidelity, it runs each evaluation in an isolated Modal Sandbox containing a commit-pinned version of its complete application stack, including document editing, databases, storage, PDF conversion, search infrastructure, and code-execution environments. Large experiments can involve 2,000 or more concurrent runs, with Modal Volumes providing shared document collections and reproducible snapshots of databases, permissions, and blobs, while image prewarming and caching reduce setup time. Legora adopted Modal VM Sandboxes after earlier container limitations required networking and filesystem workarounds, as full virtual machines allow Docker to operate like it does on standard Linux hosts. The resulting platform supports Legora’s internal benchmark for long-horizon legal tasks, guides model and agent-harness decisions, and enables engineers and agents to launch, analyze, and iteratively optimize experiments, including changes that reduced agent costs by 30 percent.
Oct 19, 2026 939 words in the original blog post.
Modal is evaluating NVIDIA’s Arm-compatible Vera CPU as a potential platform for its Sandbox service, a shift that would require substantial changes from its historically x86-only infrastructure and image-building systems. The company argues that sandbox workloads are especially difficult to schedule because hundreds of heterogeneous, bursty containers and virtual machines share hosts, creating contention and noisy-neighbor effects despite Modal’s use of burstable capacity rather than fixed peak reservations. To assess performance, Modal developed Agentless Agent Bench, a suite of 50 network-free Terminal-Bench tasks representing agent and reinforcement-learning activities such as compilation, file search, Git operations, and device emulation. While Vera’s single-task latency was only modestly better or sometimes comparable with competing CPUs, its throughput reportedly scaled much more effectively under heavy concurrent load, reaching up to twice that of a comparable Intel Granite Rapids system in early tests. Modal attributes this result largely to Vera’s memory bandwidth, which it says enables denser container packing without the throughput plateaus and latency degradation observed on other platforms, potentially improving agent responsiveness, reducing CPU and GPU idle costs, and increasing available Sandbox scale.
Oct 19, 2026 7,250 words in the original blog post.
Modal redesigned the filesystem stack for its VM Sandboxes to retain ImageFS’s lazy loading, tiered caching, image compatibility, and sub-second cold starts while overcoming performance problems caused by serving ImageFS through virtio-fs. Although virtio-fs allowed guest VMs to access the host-side filesystem, metadata-heavy operations such as chown required multiple guest-to-host round trips and host syscalls, making common workloads including package installation, code search, and recursive ownership changes substantially slower. The company instead adopted a block-device approach using virtio-blk, allowing the guest kernel’s ext4 filesystem to manage file metadata locally and asynchronously batch writes, eliminating repeated cross-VM filesystem requests. Its resulting ImageFS-blk backend constructs ext4 metadata from an image index while lazily supplying file-data blocks from ImageFS and storing modified blocks separately in a sparse file, enabling block-level copy-on-write rather than OverlayFS’s potentially expensive whole-file copying. Benchmarks showed major improvements, with 100,000 chown calls falling from 9.87 seconds under virtio-fs to 0.21 seconds, bun installation dropping from 12.5 to 1.07 seconds, and database workloads also gaining significantly. VM Sandboxes are now generally available with ImageFS-blk as their storage backend.
Oct 16, 2026 2,807 words in the original blog post.
Meta has released the weights for Muse Spark Open, an agentic coding model now available through Modal Auto Endpoints as a shared endpoint with per-token billing. The model is designed for long-horizon workflows, offering native multimodal capabilities, a one-million-token context window, multi-step planning, sequential tool use, code writing and debugging, and failure recovery. In Meta’s reported evaluations using high reasoning effort, Muse Spark Open achieved a score of 76.4 on Terminal-Bench 2.1 with the Terminus 2 agent and 60 on DeepSWE 1.1 using Meta’s preferred evaluation harness, with further methodology and settings available in the released model card.
Oct 15, 2026 113 words in the original blog post.
Modal describes integrating LiveKit Agents with open-weight or custom models on serverless GPUs to build WebRTC-based audio-video applications, demonstrated through a video avatar of Modal employee Charles Frye using BitHuman’s Expression Video Avatar and Qwen3-TTS. The setup uses LiveKit’s avatar interface and BitHuman’s self-hosted GPU container, whose launch endpoint connects the avatar to a LiveKit room. Because that endpoint returns immediately after joining while the avatar still needs persistent container access, Modal’s Sticky Sessions provide container affinity for reconnecting WebSocket clients and maintaining session capacity without an active HTTP request. A LiveKit agent creates a session when a participant joins, invokes the avatar endpoint with a session token, and closes the session after the call, while GPU snapshots reduce cold-start delays by avoiding repeated loading of large model weights.
Oct 15, 2026 323 words in the original blog post.
Modal reports that its usage has tripled since its Series C announcement five months earlier, with more than 15-fold year-over-year growth and over $1 billion in annualized run rate. The company argues that AI applications require infrastructure beyond traditional cloud tools because of scarce GPUs, large and slow-loading models, and the need to run AI-generated code in isolated environments; it has therefore built its own filesystem, container runtime, scheduler, and networking across capacity in more than 20 clouds. Modal says customers including DoorDash, Cognition, Ramp, Runway, robotics firms, and drug-discovery organizations use its platform for AI deployment, inference, video generation, robotics training, and scientific work. Its core products include Functions for executing code and Sandboxes for isolated code execution, which it says have reached a $300 million run rate, nearly 3 billion launches, and capacity for one million concurrent instances. The company is also expanding into multi-node training, reinforcement learning and post-training tools, HTTP servers, LLM-serving features, and generally available virtual machines, supported by a new $750 million Series D led by Accel at a $15.75 billion post-money valuation.
Oct 14, 2026 795 words in the original blog post.
Researchers demonstrate that large-model reinforcement learning can use heterogeneous, mixed-precision sampling fleets—including B200/B300 NVFP4, H100/H200 FP8 or BF16, and A100/RTX PRO 6000 BF16 accelerators—while training a BF16 model asynchronously on B200 GPUs. In agentic coding training of Qwen3.6-35B-A3B on repository repair tasks, uncorrected GRPO training collapsed even with homogeneous samplers because sampler–trainer mismatch grew over time, causing rising KL divergence, declining rewards, and repeated tool-use failures. The authors stabilize learning through masked importance sampling to reweight off-policy tokens, score centering to remove gradient drift, and top-p mask replay to ensure the sampler and trainer share the same candidate-token support. This combination achieved SWE-bench Pro V2 performance comparable to homogeneous BF16 B200 sampling despite an average sampler–trainer mismatch 4.7 times larger, while reducing sampler cost per update by about 15%. The findings argue that stability depends less on minimizing numerical mismatch than on keeping it bounded through off-policy correction, potentially allowing RL systems to independently optimize training and inference hardware, precision, kernels, scaling, and serving strategies.
Oct 13, 2026 3,626 words in the original blog post.
Modal has introduced generally available Endpoint Candidates, a feature for Dedicated Endpoints that lets users stage and evaluate alternative model deployments using mirrored samples of production traffic. Candidates can test new checkpoints, different models, or custom weights from Hugging Face repositories or Modal Volumes while preserving the production endpoint’s responses for users, and they provide separate URLs for quality evaluations alongside dashboard metrics and load benchmarks for latency and throughput. Once validated, a Candidate can be promoted through the existing production endpoint URL, while other Candidates remain available for further testing, with standard Modal compute pricing applying to mirrored workloads. To prevent slow or failing mirrors from affecting production, Modal’s Pingora-based proxy sends request streams through independent bounded buffers, cancels lagging mirrored requests when buffers or shared memory budgets are exceeded, and treats mirroring as best-effort. Modal plans to expand Candidate controls, comparison metrics, and automation, positioning the feature as a testing environment for automated inference optimizations targeting improved latency, throughput, and cost.
Oct 12, 2026 599 words in the original blog post.
TypeSafe AI’s manifesto emphasizes building production-focused AI rather than pursuing generalized artificial intelligence, exemplified by Jev, a “System One Model” designed for rapid, structured decision-making. Unlike autoregressive large language models that generate text sequentially, Jev accepts state and questions as inputs and returns typed choices, probabilities, and confidence scores in parallel, making it suited to applications such as customer-support routing, large-scale automation, browser use, and generative interfaces. TypeSafe developed Jev with a custom architecture, parallel sampling, and a training approach called Reinforcement Learning for Calibrated Decisions. Following its Thursday launch, Jev rapidly gained demand, reaching 15.1% of OpenRouter requests for 1,000–10,000-token contexts within a week and processing more than one trillion tokens in three days. Modal provided the serverless training and inference infrastructure needed to support this bursty, high-concurrency workload with low latency, flexible autoscaling, and control over networking and routing without requiring TypeSafe to operate its own infrastructure.
Oct 05, 2026 552 words in the original blog post.
Modal has made VM Sandboxes generally available, offering agents full Linux virtual machines through the existing Sandbox APIs by adding `runtime="vm"`. The service retains Modal’s container-oriented conveniences, including Images, fast cold starts, burstable CPU and memory, familiar filesystem and execution APIs, and large-scale concurrency, while enabling workloads that need capabilities beyond a gVisor container, such as Docker, local databases, FUSE filesystems, graphical environments, mobile simulators, and specialized kernel features. Built on Cloud Hypervisor with custom work on filesystems, lazy image loading, memory bursting, and snapshotting, the VM runtime has already supported more than 20 million VMs for early users. Linear uses it to run complete developer environments for coding agents, Legora uses it for long-running legal-agent evaluations with Docker-based services, and Snorkel uses it to simulate realistic multi-container software environments. Modal says gVisor remains the default for workloads that already work well there, while VM Sandboxes are intended as a drop-in option for users who need a more complete machine environment at the same API and pricing model.
Oct 03, 2026 1,021 words in the original blog post.
Modal announced several infrastructure features at its inaugural Runtime conference aimed at supporting AI agents, large-scale machine learning, and production inference. VM Sandboxes provide agents with isolated full Linux virtual machines capable of running Docker, databases, development servers, graphical environments, simulators, and kernel-level workloads, and are already used by companies including Linear, Legora, and Snorkel. Sandbox Sidecars add co-located but isolated containers for securely handling credentials, proxies, and trusted logic while keeping communication local and low-latency. Modal Clusters, now generally available through the `@modal.clustered` decorator, enable coordinated multi-container jobs for workloads such as training and serving trillion-parameter models, with users including Decagon, 1x, and Runway. Sticky Sessions preserve application state by routing a client back to the same running container, supporting seamless reconnections for real-time applications such as voice-model serving. Endpoint Candidates, currently in private beta for selected customers, allow teams to mirror production traffic to alternative inference configurations and evaluate quality, latency, throughput, and cost before promotion to production.
Oct 03, 2026 372 words in the original blog post.
Modal has introduced Sidecars, beta isolated containers that run alongside Sandboxes on the same host to separate trusted agent components, such as credentials, harness logic, proxies, and monitoring tools, from untrusted generated code. The company argues that conventional Sandbox isolation was designed around a broader unit of trust and leaves agents vulnerable to data exfiltration when private data, untrusted content, and external network access coexist. Existing approaches, including remote control planes and network egress controls, can improve security but may add latency or lack the flexibility needed for custom agent infrastructure. Sidecars use gVisor or VM isolation boundaries while communicating locally through an internal TCP/UDP bridge network, which Modal says is at least three times faster than communication between separate Sandboxes in the same region. They can be created dynamically through the SDK, share host CPU and memory resources with their Sandbox, support independent outbound network policies, and can be used to force Sandbox traffic through a proxy. Ramp is cited as an early user, employing Sidecars to run custom egress proxies for its Inspect coding agent, which handles more than 75% of Ramp’s merged pull requests.
Oct 03, 2026 1,380 words in the original blog post.
Modal has made its multi-node GPU offering, Modal Clusters, generally available, enabling developers to request clustered workloads through the `@modal.clustered` decorator while retaining integrations with Modal Volumes, Cloud Bucket Mounts, and Queues. The service targets large-scale AI training and inference workloads by providing on-demand GPU clusters, second-by-second billing, automated GPU and RDMA health management, and InfiniBand-based RDMA networking of up to 6.4 Tbps, with configuration for PyTorch and NCCL handled through an `rdma=True` flag. Supporting clusters required Modal to develop a gang scheduler that evaluates fleet-wide capacity and places all nodes of a workload together, rather than scheduling machines independently. Modal also added RDMA support to its secure gVisor runtime and upstreamed the changes, allowing direct GPU-memory networking without staging data through host memory. Customers including Decagon, 1X, and Runway are using the clusters for large-model fine-tuning, robotics world-model training, and distributed video-generation inference, while access is available to all Modal workspaces within plan-based GPU limits.
Oct 01, 2026 1,643 words in the original blog post.