September 2026 Summaries

Runpod’s second State of AI Compute report analyzes observed deployment choices across 1.2 million accounts in 186 countries, arguing that usage data offers a clearer view of AI adoption than surveys of intended activity. Agent-initiated deployments grew rapidly to 8.4% of users and 24% of revenue, with longer-running workloads and higher revenue per user, while AI video activity was found to focus largely on post-production such as upscaling rather than generating new footage. The report indicates that teams commonly combine frontier-model APIs with open-weight models, using fine-tuned specialist models for business-specific tasks, and that self-hosted users often adopt newly released open models within weeks, contributing to Qwen’s increased share and Llama’s decline. Resource constraints have also encouraged lower-precision, quantized model deployments, particularly for models larger than 70 billion parameters. Although AI companies remain Runpod’s largest customer segment by revenue, adoption is expanding into sectors including government, fintech, and manufacturing, with Asia identified as the fastest-growing region. Runpod also reviewed its prior GPU supply forecasts, finding that H100 growth was close to expectations, H200 growth lagged, and the broader Blackwell fleet grew faster than expected after newer hardware arrived.

Runpod Pods are GPU-backed containers that can be accessed like remote Linux servers using SSH key authentication, with official templates such as Runpod PyTorch providing SSH support by default. The process involves generating an Ed25519 key pair, adding the public key to Runpod credentials before creating a Pod, deploying a Pod with a public IP, and using the SSH-over-exposed-TCP connection rather than Basic SSH, which does not support file transfers. Users can upload and download files through command-line SCP, noting that it uses an uppercase port flag, or through FileZilla via SFTP with the Pod’s IP address, port, root username, and private key. Connection problems commonly result from using a key fingerprint instead of the public key, adding a key after Pod creation, or selecting the public rather than private key file. SCP is suited to quick or scripted transfers, while FileZilla provides a graphical option for browsing and moving files, and the same SSH credentials can also support tools such as VS Code, Cursor, or Runpod’s command-line utilities.

Runpod Enterprise extends the existing self-service Runpod platform for organizations moving AI workloads into production, offering reserved and dedicated GPU capacity, post-paid billing, contractual agreements, service-level commitments, and named technical account management without requiring teams to change platforms, APIs, or consoles. The offering consolidates Pods, Serverless, and Clusters under one organization-level agreement and monthly invoice, while adding hierarchical company-owned accounts, SSO/SAML integrations, identity-provider group mapping, role-based access controls, centralized resource ownership, cost-center tagging, billing analytics, audit trails, and organization-level credential management. Runpod cites support for SOC 2 Type II, HIPAA BAAs, GDPR DPAs, and ISO/IEC 27001 certification, alongside infrastructure spanning more than 30 GPU configurations and 32 regions. The announcement positions Enterprise as a way for teams to progress from developer-led experimentation to governed production deployments with predictable capacity, security review materials, financial controls, and ongoing technical support.

Runpod has launched Global Volumes in beta, providing elastic, region-independent storage that can be mounted by Pods in any of its data centers at `/workspace-global`. The service addresses the limitation of regional network volumes, which are tied to a single data center and can restrict GPU availability or require large data transfers when workloads move. Users can store model weights, LoRAs, tokenizers, configurations, and other infrequently written artifacts once, then deploy Pods wherever suitable GPUs are available without duplicating data or selecting storage capacity in advance. Global Volumes use object storage, scale with stored data, and charge based on usage, but they are not intended for workloads requiring high throughput, full POSIX behavior, or frequent and concurrent writes, for which regional network storage remains more appropriate. Available to all users for Pods at launch, Global Volumes can be created through Runpod’s storage interface and selected as persistent storage during Pod deployment.

Benchmarking seven embedding models across up to 24 GPU types, three input lengths, and three serving engines found that software choice can change embedding throughput by as much as 11x on identical hardware, often outweighing GPU selection. Text Embeddings Inference (TEI) generally performed best for BERT-family models such as MiniLM, e5, and bge-large, while vLLM led for Qwen3 embedding models; bge-m3 showed near parity between the engines, making operational preferences more important. For 512-token bge-m3 passages, the RTX A5000 offered the lowest measured cost at about $0.78 per billion input tokens, while H200 and H100-class GPUs delivered higher peak throughput but worse cost efficiency, supporting the conclusion that embedding workloads often favor smaller GPUs over flagship accelerators. Single-query latency for bge-m3-class models on pods was already low across tested cards, so steady bulk workloads should prioritize throughput per dollar, whereas Serverless is more appropriate for bursty traffic despite substantially higher single-request latency caused mainly by platform overhead. The results also emphasize validating vector equivalence, output modes, sequence lengths, concurrency requirements, and engine compatibility before using benchmark rankings to make deployment decisions.

Reserved GPU capacity, or a private GPU pool, provides dedicated Runpod GPU resources for organizations that have predictable baseline demand and need supply assurance, while on-demand GPUs remain suited to experiments, irregular workloads, and uncertain peaks. Customers pay for reserved capacity throughout the commitment term regardless of utilization, so sizing should be based on consistently used capacity, supported by historical usage, expected growth, GPU type and region requirements, and the business impact of unavailable compute. Reserved pools can serve Pods, Serverless endpoints, and Clusters within Runpod’s managed container environment, but they do not provide bare-metal access. When enabled, bursting allows workloads that exceed the reserved pool to use separately billed on-demand capacity, although teams should confirm available caps, queuing, and hard-stop controls based on workload needs. The guidance recommends reserving the predictable floor of demand, sharing pools across compatible workloads where possible, using on-demand capacity for variable demand, and reviewing utilization and renewal plans before a commitment expires.

Qwen3.8-Flash-Next is Alibaba’s preview serving stack for Qwen4, a 125B-parameter ultra-sparse mixture-of-experts model that activates 6B parameters per token, supports a native 262K-token context expandable to 1M with YaRN, and includes multi-token prediction, vision, Gated DeltaNet linear attention, and Qwen Sparse Attention for efficient long-context inference. Testing on Runpod with four NVIDIA H200 GPUs found that the standard vLLM worker template is currently incompatible because it uses vLLM 0.28, while this model requires version 0.29 or later and the dedicated vllm/vllm-openai:qwen38-flash-next image. Both Serverless load-balancing endpoints and GPU pods can run the model using TP4 with expert parallelism and Triton MoE support, with Serverless favoring bursty API traffic and pods better suited to debugging, interactive work, and higher sustained throughput. The deployment requires substantial storage and memory for its roughly 173 GiB FP8 checkpoint, and a critical NCCL_NVLS_ENABLE=0 environment variable prevents H200 NVLS initialization failures that otherwise cause all ranks to crash before weights load. Measured Serverless startup took about 11 minutes on a warm host, while compiled pods achieved much faster warm decoding performance, whereas eager Serverless mode reduced boot-time compilation delays but incurred slower early requests due to Triton kernel JIT compilation. The model was validated through OpenAI-compatible API calls with streaming, hidden reasoning traces, optional non-thinking mode, XML-based tool calling, and built-in vision support, demonstrating that the deployment can provide a 262K-context API after the required image, flags, hardware configuration, and cold-start considerations are addressed.

Runpod identified a Model Context Protocol failure mode in which unpaginated tool calls can return data far larger than an LLM’s context window, abruptly ending agent sessions and losing working state, particularly during consequential tasks such as deployments or migrations. Because MCP pagination provides cursors but does not currently require response-size caps, the company updated list-style tools with small default limits, hard caps, opaque cursors, and response metadata including total counts, truncation status, and next-page cursors, while enforcing limits in tool wrappers when underlying APIs lacked pagination. The account emphasizes that silent truncation is unsafe because it can cause agents to reason from incomplete data, and recommends returning references or resource URIs for large files, logs, and binary content rather than embedding payloads. It also argues that registered tool schemas and descriptions consume context before calls occur, making a smaller, more focused tool set preferable, and advocates tracking caller details and result sizes to identify oversized responses and diagnose session-ending context overflows before users encounter them.

P-Video-Edit, developed by Pruna AI, is an instruction-driven video editing model available through Runpod’s public API that can modify source videos up to 15 seconds long using a single text prompt and as many as four optional reference images. Runpod provides access without requiring users to deploy infrastructure, allowing them to test the model in its Playground or submit requests through Curl or Python’s requests package after creating an API key. Example requests demonstrate adding a reference-based roof cargo box to an SUV while preserving the rest of the scene, with configurable options including prompt upsampling, draft mode, audio preservation, and random seed selection. Completed requests return status, processing timing, cost, and a URL for the edited video, while the model is noted as incompatible with OpenAI’s SDK.