August 2023 Summaries

Truss is an updated model packaging and serving tool that was reintroduced after a year of learning and development. It aims to solve three common frustrations in the development loop, including a long dev loop for serving ML models, a need for a clear path from notebook to model server, and configuration requirements for Docker and Kubernetes. Truss achieves this by providing a simplified model.py file, a live reload workflow, and a config.yaml file that controls all attributes of the model server without requiring knowledge of Docker or Kubernetes. The tool has also been optimized for faster inference times with Stable Diffusion XL, achieving 1.92 seconds on an A100. Additionally, users can build their own open-source version of ChatGPT using Llama 2 and Chainlit, which provides a competitive large language model and an intuitive interface for building a similar application.

To optimize model inference for Stable Diffusion XL (SDXL), the author experimented with various tweaks, including reducing the number of steps from 50 to 20, setting Classifier Free Guidance (CFG) to zero after 8 steps, and using the refiner model for the final 20% of steps. Additionally, they used `torch.compile` with max-autotune to optimize the model for an A100 GPU, chose a fp16 vae and efficient attention implementation to improve memory efficiency, and deployed the optimized version of SDXL in two clicks from the model library, achieving a model inference time of 1.92 seconds on an A100. The author also notes that these optimizations can be applied to standard Stable Diffusion, achieving generation times of under a second on an A10G and under half a second on an A100.

The tutorial aims to create an open-source ChatGPT interface using Llama 2 and Chainlit, building upon the capabilities of GPT-3.5, a closed-source large language model that powers ChatGPT. The key features of this open-source version include high-quality output comparable to GPT-3.5, conversation in context, streaming output, prompt history, and security and privacy features. Users can deploy Llama-2-chat with just a few clicks from Baseten's model library, set up Chainlit, and run the cookbook to build the chat UI on localhost. The tutorial focuses on three key details that make this chatbot work: using conversation history and context, streaming Llama 2 output, and stripping unneeded characters. The project demonstrates how open-source tools can be used to create a compelling user experience similar to ChatGPT.

AudioGen is a breakthrough text-to-audio model from the AudioCraft family of models by Meta AI, capable of generating an array of sounds based on simple text inputs after being trained on publicly available sound effects. It is now available in the Baseten model library and can be deployed with just two clicks, utilizing efficient GPUs for optimal performance. Once deployed, users can run inference to generate one clip per prompt as a base64 encoded WAV file, producing impressive results such as footsteps on a wooden floor, small dog barking, and man talking with an emergency vehicle siren.

Llama 2 is a new state of the art (SOTA) in open-source large language models (OSS LLMs), offering three variants with varying sizes and capabilities, including a 4k-token context window. Stable Diffusion XL 1.0 creates high-quality images from shorter prompts, allowing users to specify exactly what they want to see without appending lengthy descriptions. Model autoscaling is now available, enabling cost-effective throughput by automatically creating and deleting replicas of the model server in response to incoming traffic, with a focus on scale-to-zero deployments that pay zero dollars when not in use.