How to Self-Host NVIDIA Nemotron 3.5 Super VL 120B on Your Own Kubernetes Cluster

NVIDIA Nemotron 3.5 Super VL 120B is an open-weights, commercially usable multimodal mixture-of-experts model that processes text, images, and video, with 120 billion total parameters but 12 billion active per token and a context window of up to one million tokens. Running its BF16 version requires roughly 240 GB of VRAM for weights alone, plus memory for KV cache, vision encoding, and operational headroom, leading NVIDIA to recommend a single node with four H100 GPUs or two GB200 GPUs; the NVFP4 version reduces weight memory to about 60 GB but should be independently evaluated for visual-task accuracy. The article advises using a Kubernetes stack consisting of compatible NVIDIA GPU drivers and device plugins, a serving engine such as vLLM or NVIDIA NIM, a GPU-enabled Deployment configured for intra-node tensor parallelism, and an authenticated OpenAI-compatible endpoint. It emphasizes keeping all GPUs for a replica within one NVLink-connected node, staging large model files to avoid slow restarts, providing sufficient shared memory and startup time, monitoring queue depth and GPU cache metrics, and avoiding driver, CUDA, or inference-engine version mismatches. Self-hosting can support data residency, customization, predictable latency, and high sustained throughput, but its economics depend heavily on utilization because an eight-H100 cloud node can cost roughly $40,000 per month when continuously running; hosted APIs are generally more economical for low or bursty demand, while scale-to-zero pools, committed-use discounts, spot capacity, and quantization can reduce self-hosting costs.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.