Turbocharging Llama 2 70B with NVIDIA H100

Perplexity benchmarked Llama 2 70B inference using NVIDIA A100 and H100 GPUs with TensorRT-LLM, testing the effects of GPU architecture, fp8 and other quantization methods, tensor parallelism, and batch size on latency, throughput, memory use, and accuracy. Using synthetic workloads with 1,024 input and 512 output tokens, the experiments found that H100 GPUs substantially outperformed A100s, delivering 54% of the latency and 184% of the throughput under identical fp16 configurations, with native fp8 improving these results to 49% latency and 202% throughput relative to A100. Accuracy testing on WikiText showed less than 1% perplexity change for fp8 and 8-bit SmoothQuant, while 4-bit weights caused a roughly 7% change. Higher batch sizes increased throughput but also latency and KV-cache memory demands, while lower tensor parallelism improved per-GPU throughput but could make responses slower. The highest-throughput setup, H100 fp8 with tensor parallelism of two and batch size 128, reached 767 output tokens per second per GPU, or 460% of the A100 baseline, though its latency was likely too high for production; alternative configurations offered more balanced performance. Based on the findings, Perplexity plans to transition from A100 to H100 hardware and explore structured sparsity and int4 precision for further gains.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.