November 2023 Summaries

Perplexity announced the public release of its pplx-7b-online and pplx-70b-online models through the pplx-api and Perplexity Labs, positioning them as internet-connected LLMs designed to improve response freshness and factuality. Built on Mistral-7B and Llama 2-70B, the models combine web search, indexing, crawling, ranked website excerpts, and fine-tuning to answer time-sensitive questions beyond their original training data. Perplexity evaluated the models against GPT-3.5 Turbo and Llama 2-70B Chat using human pairwise comparisons across helpfulness, factuality, freshness, and overall preference, with evaluators able to verify claims through online search. According to the company’s Elo-score and win-rate results, its online models matched or outperformed the comparison models, especially for current and accurate information. The release also makes its chat models generally available, introduces usage-based API pricing, provides Pro users with a recurring $5 monthly API credit, and promotes infrastructure powered by NVIDIA H100 GPUs.

Perplexity benchmarked Llama 2 70B inference using NVIDIA A100 and H100 GPUs with TensorRT-LLM, testing the effects of GPU architecture, fp8 and other quantization methods, tensor parallelism, and batch size on latency, throughput, memory use, and accuracy. Using synthetic workloads with 1,024 input and 512 output tokens, the experiments found that H100 GPUs substantially outperformed A100s, delivering 54% of the latency and 184% of the throughput under identical fp16 configurations, with native fp8 improving these results to 49% latency and 202% throughput relative to A100. Accuracy testing on WikiText showed less than 1% perplexity change for fp8 and 8-bit SmoothQuant, while 4-bit weights caused a roughly 7% change. Higher batch sizes increased throughput but also latency and KV-cache memory demands, while lower tensor parallelism improved per-GPU throughput but could make responses slower. The highest-throughput setup, H100 fp8 with tensor parallelism of two and batch size 128, reached 767 output tokens per second per GPU, or 460% of the A100 baseline, though its latency was likely too high for production; alternative configurations offered more balanced performance. Based on the findings, Perplexity plans to transition from A100 to H100 hardware and explore structured sparsity and int4 precision for further gains.