March 2026 Summaries

A recent study, "When Does Divide and Conquer Work for Long Context LLM?" presented at ICLR 2026, explores the effectiveness of a "Divide & Conquer" strategy to improve performance on long-context tasks for large language models (LLMs). Instead of relying on a single, powerful model to process extensive data, the framework suggests using smaller models that divide the task into manageable chunks, which are then individually processed and aggregated to form a cohesive output. This method helps mitigate issues like model noise, task noise, and aggregator noise, which can degrade performance as context length increases. Crucially, it was observed that smaller models using this strategy can outperform models like GPT-4o in single-shot scenarios, particularly in tasks with moderate cross-chunk dependencies, such as question-answering, retrieval, and summarization. The approach offers practical benefits, including reduced costs, faster processing due to parallel execution, and easier tuning of the model's chunk size. However, the strategy is not universally applicable, as its effectiveness diminishes in tasks requiring comprehensive, interconnected context across the entire input, where a single, powerful model may still be necessary.

The research paper "When Does Divide and Conquer Work for Long Context LLM?" explores a novel framework that uses a "Divide & Conquer" approach to enhance the performance of smaller language models on long-context tasks, potentially surpassing the capabilities of larger models like GPT-4o in single-shot scenarios. The study reveals that as context length increases, models experience superlinear growth in confusion, termed "Model Noise," while "Task Noise" arises from dependencies across text chunks, and "Aggregator Noise" affects the integration of partial answers. By strategically dividing tasks into manageable chunks and employing smaller models to handle them in parallel, the framework offers benefits such as reduced costs, faster processing, and easier tuning, proving effective in tasks like retrieval, QA, and summarization, although not universally applicable, especially in cases where significant cross-chunk dependencies exist.

Together AI has introduced advancements in its Together Fine-Tuning service, which now supports fine-tuning for tool calls, reasoning, and vision-language models (VLMs) to enhance reliability and performance in AI workflows. Tool call fine-tuning ensures structured, production-grade execution by validating schemas and improving inference reliability, while reasoning fine-tuning allows models to learn domain-specific logic through thinking traces. The service also offers native support for vision-language models to better interpret domain-specific visual data, with options for joint training of vision and language layers. To accommodate large models, Together AI has upgraded its infrastructure to efficiently handle models with up to 1 trillion parameters, achieving significant throughput improvements. This update also includes features like job cost and time estimations to aid planning and budgeting, enabling faster iterations and reduced costs for AI teams.

Mamba-3 is a newly developed state space model (SSM) that prioritizes inference efficiency, contrasting with Mamba-2's focus on training speed. Key enhancements include a more expressive recurrence formula, complex-valued state tracking, and a multi-input, multi-output (MIMO) variant, which collectively enhance accuracy without increasing decoding latency. Mamba-3 outperforms its predecessor, Mamba-2, as well as other models like Gated DeltaNet and Llama-3.2-1B, particularly in prefill and decode latency at a 1.5B scale. These improvements are inspired by traditional control theory and leverage advanced kernel development using Triton, TileLang, and CuTe DSL to optimize hardware performance. The model's architecture has been updated to align with modern language models, incorporating new components such as RoPE and MIMO projections, enhancing its capabilities without compromising speed. Mamba-3 is particularly effective in language modeling tasks while maintaining competitive performance in retrieval tasks with its fixed-size state. The ongoing shift in focus from training to inference in large language models prompted the development of Mamba-3, aiming to optimize the quality-efficiency frontier by allowing better models to run faster. The open-sourcing of the kernels encourages further exploration and development based on Mamba-3's architecture.

Together AI participated in NVIDIA GTC 2026, highlighting the evolving landscape of AI systems becoming more open, agentic, and production-ready. Key announcements included the launch of NVIDIA Dynamo 1.0 for generative and agentic inference, which Together AI is integrating into its inference stack. They also unveiled NVIDIA NemoClaw, an open-source stack for running always-on assistants, and showcased NVIDIA Nemotron 3 Super, a model for multi-agent workflows with high-performance reasoning capabilities. Additionally, the NVIDIA Parakeet TDT 0.6B V3 ASR model was introduced for building production-ready voice agents. Together AI conducted sessions on AI research, production inference, and hosted various events to discuss future AI infrastructure.

Together AI has launched a comprehensive platform designed to enhance the development and deployment of real-time voice agents by unifying speech-to-text (STT), language model (LLM), and text-to-speech (TTS) capabilities within a single cloud infrastructure. This approach reduces latency by keeping data transfers within one cloud environment, thus eliminating the delays associated with multiple vendor hops. The platform supports leading voice models such as Cartesia Sonic-3 for TTS and Deepgram for STT, offering developers flexibility in choosing and configuring models for various use cases. Together AI's architecture ensures reliability and security for enterprise deployments, adhering to standards like SOC 2 Type II and HIPAA, while maintaining zero data retention. Additionally, the platform simplifies operational complexity by providing a unified API, authentication, and billing system, which reduces overhead and enhances cost predictability.

AI infrastructure has seamlessly transitioned into production environments as teams manage workloads that rapidly scale from single-node prototypes to distributed systems utilizing hundreds of GPUs. This shift necessitates advanced management tools to handle unpredictable traffic spikes and potential hardware failures, which can disrupt training processes. To address these challenges, Together GPU Clusters introduces significant enhancements such as autoscaling, Role-Based Access Control (RBAC), full-stack observability, and self-serve node repair, integrated into the core cluster experience. These features, driven by the Kubernetes Cluster Autoscaler, allow for dynamic GPU capacity management, ensuring performance without excessive costs. The platform's observability tools, including a dedicated Grafana instance, provide comprehensive telemetry for performance monitoring and cost efficiency. By incorporating robust access controls and active health checks, Together GPU Clusters enable organizations to confidently transition from experimental to operational AI systems, supporting diverse internal stakeholders and aligning resource allocation with real-time demands.

NVIDIA's Nemotron 3 Super is a 120-billion-parameter hybrid AI model designed for complex reasoning and multi-agent orchestration, combining Transformer and Mamba architectures. It is optimized for high-throughput inference workloads and is available on Together AI's Dedicated Inference platform, which handles infrastructure requirements such as GPU provisioning for users. The model features several innovations, including a Hybrid Mixture-of-Experts architecture that efficiently manages active parameters, a 1-million-token context window for processing large data sets, and multi-token prediction to accelerate output generation. It is particularly suited for applications involving large document analysis, multi-step planning, and agent coordination. Nemotron 3 Super is trained using reinforcement learning and synthetic data, with open weights and customization options for engineering teams, ensuring flexibility to adapt the model to specific needs.

Together Research announced several advancements at the AI Native Conf, showcasing innovations like FlashAttention-4, a kernel co-design that significantly enhances the performance of large-scale language models on NVIDIA GPUs, achieving faster processing at lower costs. Their Megakernel implementation, tailored for real-time voice agents, dramatically improved performance metrics by optimizing the entire model in one kernel. The introduction of together.compile automates kernel optimization, boosting production efficiency for video models. The new Reinforcement Learning API provides teams with control over RL training configurations, enhancing rollout efficiency. ThunderAgent overcomes challenges in agentic workflows by treating them as cohesive scheduling units, resulting in substantial throughput improvements. ATLAS-2, a speculative decoding method, continuously updates speculator models in real-time, maintaining performance as traffic patterns shift. Additionally, Cache-aware prefill–decode disaggregation (CPD) optimizes long-context inference, achieving higher throughput by managing cache usage effectively. Together's approach emphasizes the synergistic relationship between research and production, aiming to expand AI infrastructure capabilities for demanding applications.

FlashAttention-4 is an advanced algorithm and kernel co-design that optimizes the performance of attention mechanisms on Blackwell B200 GPUs, particularly addressing the challenges posed by asymmetric hardware scaling where tensor core throughput significantly outpaces other resources. By implementing novel pipelining and scheduling techniques, FlashAttention-4 maximizes the overlap between matrix multiplications and other resource bottlenecks such as the softmax exponential computation and shared-memory traffic, ultimately achieving up to 1605 TFLOPs/s on B200 with BF16, surpassing the performance of cuDNN and Triton. Key innovations include a software emulation of the exponential function to alleviate bottlenecks, a new tile scheduler for load balance, and a backward pass design that reduces shared-memory traffic and atomic operations for deterministic execution. Implemented in CuTe-DSL, FlashAttention-4 demonstrates significant performance improvements in attention benchmarks, offering faster computation for large sequence lengths while maintaining high utilization of the Blackwell architecture's capabilities.

Together AI has unveiled a refreshed brand identity to better reflect its mission of promoting open innovation and supporting AI-native builders in creating groundbreaking products. The company emphasizes the importance of open systems over closed ones, allowing for faster integration of new AI models and technical breakthroughs. Together AI provides a comprehensive platform that includes open-source models, advanced systems research, and a collaborative developer experience, all aimed at improving performance and reducing costs for AI-native developers. The new visual identity, designed in collaboration with Pentagram, embodies the connectedness of research, infrastructure, and innovation, promoting a shared scale and open knowledge. Together AI aims to empower small teams to make significant advancements, turning innovative ideas into scalable realities.

Running large language models (LLMs) in production requires more than just deploying a model; it involves optimizing inference infrastructure to ensure scalability and efficiency. Key components include optimized kernels that maximize GPU utilization, quantization strategies that reduce costs while preserving quality, speculative decoding techniques to lower latency, and infrastructure designs that handle real-world traffic patterns. The competitive AI landscape demands rapid responses, reshaping infrastructure needs as inference costs become a significant part of operational expenses. Companies must optimize time to first token and throughput to maintain sustainable unit economics and reduce costs per request. Successful companies rethink how models, runtimes, and hardware interact, as off-the-shelf frameworks often leave performance potential untapped. Understanding inference economics is crucial, as it forms the foundation of a company's profit and loss statement. Meeting customer expectations for quick responses while maintaining business fundamentals is essential, as latency directly affects user experience and product feasibility. Speculative decoding allows models to generate text faster by proposing multiple tokens for verification, optimized kernels enhance GPU performance, and custom or adaptive speculators improve speed and cost-efficiency. These strategies collectively enhance AI product competitiveness by reducing latency and operational costs.

Optimizing training workloads on GPU clusters involves strategic planning and validation to enhance throughput, reliability, and cost-efficiency for machine learning engineers, infrastructure specialists, and MLOps teams. This approach requires careful orchestration of compute, storage, and data pipelines, particularly for training modern machine learning models like large language models and multimodal systems. Effective practices include cluster planning with appropriate GPU selection, data placement near GPU nodes to reduce latency, and choosing suitable orchestration systems like Kubernetes or Slurm based on workload needs. Ensuring software stack compatibility and conducting pre-training validations such as access verification and hardware health checks are crucial for avoiding runtime errors and performance degradation. Optimization techniques such as workload profiling, data pipeline optimization, and minimizing network overhead are essential for efficient GPU utilization. Monitoring and observability, along with failure recovery strategies, are also vital to maintaining operational efficiency and reducing downtime. Together AI’s infrastructure platform supports these efforts by offering instant cluster provisioning and pre-configured software stacks, facilitating streamlined training pipelines.