August 2024 Summaries

TEAL (Training-Free Activation Sparsity in Large Language Models) presents a simple training-free approach to activation sparsification, achieving 40-50% model-wide activation sparsity with minimal degradation. This allows for significant speedups in inference, particularly in single-batch decoding, with improvements ranging from 1.53x to 1.8x wall-clock speedups. TEAL targets the entire model, including tensors not previously sparsified, and outperforms existing methods like CATS by optimizing sparsity levels at the transformer block level. Additionally, TEAL demonstrates compatibility with quantization techniques, offering a promising direction for efficient LLM inference. The approach is designed to be flexible and adaptable to various applications, particularly in resource-constrained edge settings.

Together Rerank API offers a new serverless endpoint for integrating reranker models into enterprise applications, with exclusive access to Salesforce's LlamaRank model, which outperforms leading competitors like Cohere Rerank v3 and Mistral-7B QLM. The API provides a seamless developer experience, allowing users to build and manage their entire generative AI lifecycle from training and fine-tuning to inference, using both open and proprietary models. It supports long document sizes up to 8,000 tokens in length and can handle semi-structured data such as JSON, email, tables, and code. The API is compatible with Cohere Rerank, enabling easy experimentation with different models for RAG applications, and provides a flexible solution for enhancing search accuracy and reducing costs by filtering out irrelevant documents that are passed to language models during Retrieval Augmented Generation (RAG).

At Together AI, they have developed a systematic approach to acceptance testing for GPU clusters designed to guarantee reliability and performance for demanding AI/ML workloads. This process involves configuring the cluster's hardware environment, stress testing and benchmarking individual subsystems and components, validating NVLink and NVSwitch communication, testing network configurations, measuring storage performance, running reference tasks tailored to customers' use cases, and continuously monitoring for hardware failures using tools like Telegraf. By adopting this comprehensive approach, companies can navigate the complexities of GPU clusters and ensure their infrastructure is stable and reliable, supporting top-tier computational resources and delivering expected end-to-end performance.