December 2023 Summaries

In 2023, the field of open-source machine learning (ML) models saw significant advancements. Mixtral 8x7B and Stable Diffusion XL set new standards for LLMs, while tools like LangChain and retrieval augmented generation became mainstream. The year also brought improvements to inference speed using TensorRT-LLM and int8 quantization with Mixtral. Additionally, Playground v2 was released, a text-to-image model that can create consistent, stylized images. ComfyUI projects were also showcased in a tutorial on deploying inference pipelines as API endpoints, enabling wider sharing and integration of image generation models. New developments are expected in 2024, but the current year's progress demonstrates the growth and maturity of open-source ML models.

Mixtral 8x7B is an LLM that produces results comparable to larger models like Llama 2 70B and GPT-3.5, but with fewer parameters and enabling faster inference. Using TensorRT-LLM and quantizing the model to int8 achieves important performance milestones while using only a single A100 GPU. Mixtral's mixture of experts architecture uses only 12.9B parameters during inference, and its performance is better for individual requests than larger models. However, batching inference does not take full advantage of this architecture, resulting in decreased throughput. Quantizing the model to int8 cuts inference cost in half while preserving quality with a minimal increase in perplexity. The use of TensorRT-LLM unlocks faster single-request and batched inference performance, making Mixtral suitable for a wide range of use cases.

Playground v2 is a new text-to-image model that rivals Stable Diffusion XL (SDXL) in speed and quality, with a more opinionated aesthetic similar to AAA video games. Trained from scratch, Playground v2 has consistent output images and is comparable in inference speed to SDXL. While SDXL offers a wider variety of images, Playground v2 wins head-to-head comparisons approximately 70-75% of the time depending on the prompt set used. The model can be deployed in just two clicks from the model library and requires less GPU power than SDXL for inference.

To deploy an image generation pipeline built with Comf UI behind an API endpoint, it's necessary to first build and export the project in a specific format using Truss. This involves setting up a Baseten account and API key, installing Truss, cloning the prepared project, deploying the model to Baseten, and linking the workflow JSON file and model weights. Once deployed, users can call the model endpoint with parameters such as positive prompt, negative prompt, controlnet image, and seed, and retrieve the generated images in base64 format. The process allows developers to share and use Comf UI pipelines in applications.