Serving world models: Request latency and node throughput tradeoffs

Benchmarking NVIDIA Cosmos 3 Super text-to-video generation through vLLM-Omni on eight-GPU NVIDIA HGX B200 and H200 nodes found that dividing a node into more independent replicas increased validated video output but substantially raised per-request latency. On the B200, eight single-GPU replicas generated 589.4 valid video-seconds per node-hour, 42.5% more than one eight-GPU replica’s 413.6, reducing node cost per video-second by 30% at the same hourly price, while mean latency increased from 68.5 to 378.1 seconds. The study defined throughput as successfully validated video duration per node-hour and controlled the model revision, software container, prompt, seeds, video settings, request volume, timing boundary, and validation checks; all 408 generated videos passed technical validation. Results followed the same pattern on H200 systems, though with different absolute performance, and increasing requests in flight per replica produced little additional throughput while raising latency significantly. The findings suggest that eight single-GPU replicas best suit batch synthetic-data workloads, whereas a single eight-GPU replica is preferable for interactive use cases requiring faster clip completion, with intermediate layouts offering tradeoffs based on latency requirements.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.