October 2024 Summaries

QTIP (Quantization with Trellises and Incoherence Processing) is a new weight-only LLM post-training quantization method that achieves state-of-the-art quality and inference speed. It compresses model weights using trellis coded quantization, which significantly improves over QuIP#'s quality while being 3X faster than unquantized models. QTIP builds upon the incoherence processing framework introduced by QuIP and uses trellis coded quantization to achieve lower distortion on i.i.d Gaussian sources compared to vector quantization. The bitshift trellis and compute-based codes enable fast decoding for weight-only quantization, making QTIP practical for memory-bound inference settings.

LoLCATs (Low-rank Linear Conversion via Attention Transfer) is a new approach for quickly creating subquadratic LLMs from existing Transformers, focusing on accelerating models and creating fast models more efficiently. The method involves replacing softmax attentions with linear attentions trained to approximate their softmax counterparts ("attention transfer") and adjusting the model by only adjusting with parameter-efficient finetuning (e.g., low-rank adaptation). LoLCATs allows for state-of-the-art linearized quality, drastically reduces linearizing costs, and scales up to 70B and 405B LLMs.

BlinkShot is an open-source app that generates images from text in real-time using Flux and Together AI. The app uses React Query to generate images as the user types, and it can be built with Next.js, Shadcn, and React Query. The Turbo endpoint for the FLUX.1 [schnell] model from Black Forest Labs is used to create images quickly. Users can install Together's node SDK to generate an image with Flux [schnell]. Debouncing API requests can be implemented using the @uidotdev/usehooks library, and the steps option of images.create controls the number of generation steps for higher quality images. The seed option enables deterministic image generations by reproducing a generation for a given prompt.

The text discusses a new method called ColPali for indexing and embedding document pages directly, bypassing the need for complex extraction pipelines. Combined with cutting-edge multimodal models like Llama 3.2 vision series, ColPali enables AI systems to reason over images of documents, enabling a more flexible and robust multimodal Retrieval Augmented Generation (RAG) framework. The traditional approach involves OCR for scanned text, language vision models to interpret visual elements like charts and tables, and augmenting text and descriptions with structural metadata such as page and section numbers. ColPali directly indexes and embeds document pages as images, retrieving based on visual semantic similarity. It can handle complex document formats efficiently and accurately while preserving the original document layout. The new series of Llama 3.2 vision models use a technique called visual instruction tuning to imbue LLMs with vision capabilities, allowing them to process images and complete multimodal RAG workflows.

Together AI now supports the FLUX model family, including access to the latest high-performance premium model FLUX1.1[pro] and free access to FLUX.1 [schnell]. This collaboration with Black Forest Labs brings state-of-the-art image generation models to the Together Platform with the best performance, price, and developer experience. Developers can now leverage these models for their applications through the platform's APIs or playground.

Zomato, an Indian food delivery company, built an AI-powered support bot called Zia to improve customer satisfaction and scale their support system. The bot uses large language models (LLMs) and a response generation pipeline to analyze conversations, identify required data, and generate contextually aware prompts. To optimize for latency, scalability, and cost efficiency, Zomato leveraged Together AI's optimized Llama models and scalable infrastructure. The switch to open-source models reduced costs and improved performance, allowing the bot to handle over 1,000 messages per minute during peak periods like Mother's Day. The implementation resulted in a 2x improvement in customer satisfaction scores, a 75% reduction in response times, and cost efficiency gains of 20%. Zomato continues to evaluate their system with human-in-the-loop quality checks and LLM-as-judge grading, and is exploring new areas such as multimodal models and augmenting human support with AI.