February 2024 Summaries

NVIDIA has released new improvements in February 2024, focusing on model performance across four key factors: latency, throughput, quality, and cost. The company now offers model inference on H100 GPUs, which feature exceptional performance for running ML models due to their high tensor compute, memory bandwidth, and VRAM. This results in a significant reduction in cost for running high-traffic workloads. Additionally, NVIDIA has optimized Stable Diffusion XL with TensorRT, achieving 40% lower latency and 70% higher throughput on H100 GPUs compared to a baseline implementation. The company has also introduced SDXL Lightning, which generates images in under one second per image, while QwenVL is an open-source visual language model that combines vision and language capabilities. Furthermore, NVIDIA's refreshed billing dashboard provides daily insights into usage and spend, offering improved visibility for users.

SDXL is a text-to-image model that can generate images with high quality and flexibility. It uses a modular architecture composed of four major components: CLIP, UNet, Refiner, and VAE. The UNet model is the main component of SDXL and runs iteratively in inference steps to create an image representation in latent space. Optimizing the performance of SDXL involves individually optimizing each component in the pipeline using NVIDIA TensorRT, a software development kit for high-performance deep learning inference. The optimization process includes exporting the model pipeline to ONNX, making an optimized engine for serving each sub-model within SDXL, and deploying the optimized models as API endpoints. With TensorRT, SDXL achieves up to 40% lower latency and 70% higher throughput than the unoptimized model on the same hardware, making it viable for high-latency and cost-sensitive use cases. The techniques used can be applied to similar image generation pipelines, including SDXL Turbo, which generates images with even higher quality but at a slightly lower speed.

GPU utilization is crucial for model inference as it directly affects the cost of serving high-traffic workloads. A high GPU utilization means fewer GPUs are needed, saving on costs. Measuring GPU utilization involves considering compute usage, memory usage, and memory bandwidth usage. Increasing batch sizes during inference can improve utilization by increasing throughput while managing trade-offs with latency. Switching to more powerful GPU types can also save costs. Tracking GPU utilization in the Baseten workspace provides insights into real-world usage effects on utilization.

Llama 3.1 70B Instruct is considered the best overall open source LLM, offering a large context window and excellent retrieval benchmarks, along with strong code generation and mathematical reasoning capabilities. It has some limitations, such as only supporting eight languages and having a custom commercial license that applies to fine-tuned derivatives. On the other hand, Llama 3.1 405B is a powerful big LLM that rivals heavyweights like GPT-4o, with a massive 128k-token context window for retrieval-augmented generation and tool use. However, it requires significant computational resources and has a custom commercial license. The best small LLM under 7 billion parameters is Phi 3 Mini, which offers excellent output quality and permissive MIT licensing. Mixtral 8x7B is another great option, with high output quality and efficient inference on A100 GPUs. Zephyr models are the best aligned chat LLMs, offering helpful assistant behavior and supporting ChatCompletions-style roles out of the box. Code Llama is the best model for code generation, with strong performance on evaluation benchmarks and four sizes to choose from. Llama 3.1 is a great option for fine-tuning projects, with flexible base models and explicit licensing for derivatives. Ultimately, the best open source LLM depends on specific use cases and requirements, and it's recommended to experiment and find the right model for each task.

NVIDIA's H100 GPUs offer the most powerful processors on the market but running inference on ML models takes more than raw power. To get the fastest time to first token, highest tokens per second, and lowest total generation time for LLMs and models like Stable Diffusion XL, developers turn to TensorRT, a model serving engine by NVIDIA. By serving models optimized with TensorRT on H100 GPUs, developers unlock substantial cost savings over A100 workloads and outstanding performance benchmarks for both latency and throughput. The H100's increased memory bandwidth has a direct impact on an LLM's performance, resulting in double the throughput vs A100 and 2x improvement in latency with constant batch size for Mistral 7B. Additionally, the H100's faster time to first token from better Tensor compute and 3x better throughput at increased batch sizes make it an attractive option for demanding ML workloads. The H100 offers substantial advantages over A100s, including 1.6x more memory bandwidth and 989.5 teraFLOPs of fp16 tensor compute, but running inference with TensorRT/TensorRT-LLM yields even bigger improvements over the A100 than the stat sheet would suggest.