August 2024 Summaries

The article discusses the best open-source image generation models available today. These AI models can create detailed, realistic images from simple text prompts. The top contenders include FLUX.1 by Black Forest Labs, Stable Diffusion 3 by Stability AI, SDXL Lightning by Bytedance, and Playground 2.5 Aesthetic 1024x1024. Each model has its unique strengths and weaknesses, such as output quality, prompt adherence, typography capabilities, inference speed, and commercial use permissions. The article also provides guidance on how to get more control over model output, build complex image generation pipelines, and deploy these models for various applications.

Medusa is a technique for generating multiple tokens per forward pass during LLM inference, which can double the tokens per second of an LLM deployment. After training and validating Medusa heads, additional decoding heads grafted onto the base model, Medusa can be used in production by deploying the modified LLM using TensorRT-LLM. In a benchmark, Medusa was found to double the tokens per second running Llama 3 8B on an A100 in FP16 with no other major optimizations in place. However, it is crucial to validate output quality before deploying a model with Medusa to production.

The recent "Llamathon" hackathon at South Park Commons saw teams pushing the boundaries of Meta's new Llama 3.1 family of open-source models, with two winning projects: TestNinja and VibeCheck. TestNinja used a combination of traditional software engineering and AI engineering to create an LLM-powered test generation tool that leverages deterministic Python code to extract relevant context for test generation, while VibeCheck automated the manual process of generating early-stage design concepts using Llama 3.1's inherent design intelligence, reducing time spent on mood boards by designers. Both projects utilized Baseten's free model hosting and technical support to deliver impressive results.

Baseten Self-hosted is a self-managed solution that allows companies to run AI model inference on their own cloud, providing granular control over data locality, compliance with organizational or industry standards, and meeting specific performance and latency requirements. It leverages Truss for packaging and deploying models, ensuring reliable, fast, and convenient deployment of models in the customer's VPC without touching Baseten's premises. The solution is ideal for companies requiring strict security measures, custom hardware utilization, or optimal use of existing resource investments, offering zero-downtime deployments and regular updates from Baseten's engineering team. It supports compound AI systems, meets specific data residency requirements, utilizes existing cloud credits, and provides saved engineering time by partnering with industry experts.

Compound AI systems integrate multiple models and processing steps to form an integrated workflow, offering flexibility, performance, and cost efficiency over monolithic workflows. They are becoming increasingly popular in various industries as new AI models set records across different domains, enabling the creation of more complex tasks such as multi-modal chatbots, phone call agents, video editors, and healthcare diagnostic support systems. The modularity of compound AI systems allows developers to iterate on individual components, reuse them in different pipelines, and make cost-effective design options. However, building, optimizing, and serving these systems in production can be challenging due to coordination logic requirements, intra-cloud roundtrips, and monolithic server limitations. To address these challenges, open-source frameworks like Chains are emerging, providing tools for building and orchestrating compound AI systems with ease.

The TensorRT-LLM Engine Builder is a tool that automates the process of building optimized model serving engines for open-source and fine-tuned large language models (LLMs) in minutes, replacing hours of manual work previously required. It uses the TensorRT-LLM performance optimization toolbox to create efficient inference servers with low latency and high throughput, compatible with over 50 LLMs and similar models. The engine builder is built into Truss, an open-source model packaging framework, and provides full control over the model server, including autoscaling, logging, and metrics, as well as secure and compliant inference. It can be used to build inference engines maximized for latency, throughput, cost, or a balance thereof, depending on the user's goals, such as supporting concurrent requests or minimizing latency.