July 2026 Summaries

The Together AI platform offers a model inference solution that enables autoscaling of deployments based on metrics the inference engine comprehends, such as in-flight requests, time to first token (TTFT), GPU utilization, and token throughput. Users can configure replica bounds, choose appropriate metrics, and fine-tune scaling windows to optimize for cost and performance under varying traffic conditions. This approach is crucial for effectively handling peaky traffic while minimizing latency impacts. The platform's autoscaling mechanism involves a continuous feedback loop that adjusts replica counts based on observed metrics relative to targets, with timing windows that manage how quickly replicas are added or removed. Different autoscaling policies can be applied depending on specific deployment needs, such as concurrency-driven, SLO-driven, or efficiency-driven metrics, each with distinct implications for system behavior and costs. The document highlights the importance of selecting the right metric to avoid under-provisioning, which can lead to significant latency spikes, and over-provisioning, which incurs unnecessary expenses. It also emphasizes the need for an intuitive understanding of traffic patterns to set optimal scaling windows and provides guidance on refining autoscaling policies through empirical observations and adjustments.

Together AI has announced a strategic partnership with Moonshot AI, focusing on advancing open-source large-scale Mixture-of-Experts (MoE) architectures. This collaboration makes Together AI the launch platform for Moonshot's model releases, starting with the Kimi K3, a 2.8 trillion parameter sparse MoE model designed for long-horizon coding and knowledge-intensive tasks. Kimi K3 introduces innovations like Kimi Delta Attention and Attention Residuals, offering improved training efficiency and performance, competing with proprietary systems. The partnership provides developers with day zero access to these models via Together AI’s infrastructure, featuring options like serverless deployment, provisioned throughput, and dedicated inference with SLA-backed products, ensuring predictable pricing and no data retention. Developers can further customize and post-train Kimi models using Together AI’s Python SDK, enabling seamless transitions from experimentation to production. This alliance aims to provide developers with scalable, flexible, and high-performance model deployment without vendor lock-in, aligning with the needs of companies like Cursor and Y Combinator that rely on Together's infrastructure for production traffic.

ThunderAgent is a high-throughput system designed to optimize agentic inference by introducing a novel program abstraction for scheduling requests, resulting in significant throughput and latency improvements. Unlike traditional inference engines that treat each language model call as an independent request, ThunderAgent manages entire workflows as schedulable programs, tracking their execution phases, memory usage, and node assignments. This approach alleviates key inefficiencies such as KV cache thrashing by pausing low-priority workflows during high memory pressure and resuming them on nodes with available capacity through a global waiting queue. The system demonstrates up to 2.5× higher throughput on single nodes and 2.4× speedup on multi-node clusters, with near-linear scaling across GPU nodes. ThunderAgent integrates seamlessly with existing setups, requiring minimal adaptations, and has been adopted in various open-source frameworks. Its ability to balance loads across nodes and effectively manage memory makes it a promising foundation for future agentic inference systems.

Together AI's Dedicated Model Inference platform integrates endpoints, deployments, and configs, which are tied together by a capacity-aware traffic split to facilitate model operations such as rollouts, A/B tests, and shadow experiments without downtime. An endpoint has a stable identity that applications use to call models, while deployments link specific models to configs and manage replicas with autoscaling policies. Configs specify the model's operational parameters, including engine type, GPU usage, and optimization profiles, and are immutable, ensuring consistent deployment behavior. Traffic routing is managed through weights assigned to deployments, allowing for proportional traffic flow based on capacity, which adapts to changes in replica counts. The platform allows for various traffic management strategies such as A/B testing and canary deployments, with traffic splits, A/B member percentages, and canary step percentages serving distinct roles in the routing process. The Together AI platform automates deployment setup and traffic routing, ensuring that new deployments do not receive traffic until specified in the endpoint's traffic split. Users can choose from certified profiles for their models, optimizing for latency, throughput, or a balance of both, depending on the use case, and can conduct tests to determine the best configuration for their traffic needs.

The comparison between Kimi K3 and GPT-5.6 Sol on the DeepSWE benchmark highlights their distinct strengths and weaknesses, with Kimi K3 excelling in cost-effectiveness and pass@k metrics while GPT-5.6 Sol demonstrates higher reliability and single-attempt quality. GPT-5.6 Sol leads slightly in pass@1 with a 72.7% success rate compared to Kimi K3's 68.5%, yet Kimi K3 surpasses Sol in pass@2 and pass@4, at a considerably lower cost per rollout (\$4.65 compared to Sol's \$8.37). Kimi K3 achieves 14.7 solved tasks per \$100, making it about 2.8 times more cost-efficient than Sol. Despite Sol's steadiness and higher reliability (84.5% with tasks solved four-for-four), Kimi K3 offers broader coverage with 89.4% of tasks solved at least once across four tries. The two models exhibit a 0.46 correlation in task performance, indicating they succeed and fail differently, making a combined routing strategy between them advantageous. This approach, using a Kimi-first cascade with escalation to Sol when necessary, covers 108 of 113 tasks and achieves an 85.6% success rate while maintaining cost efficiency. The analysis suggests that the optimal strategy for many teams is to leverage both models, exploiting their complementary strengths.

Kimi K3, an open-weight model, offers a compelling alternative to Claude Fable 5 in the DeepSWE benchmark by providing similar quality at a significantly lower cost, making it a strong value choice for tasks that benefit from multiple attempts. While Claude Fable 5 demonstrates higher single-attempt reliability and solves more tasks consistently, Kimi K3 excels in broader task coverage and gains an advantage in metrics such as pass@2 and pass@4, indicating it eventually solves more tasks with additional attempts. Despite being a third of the cost, Kimi K3 delivers 2.8 times more solved tasks per dollar compared to Claude Fable 5, highlighting its cost-effectiveness for high-volume workloads. Both models show a high task similarity, with Kimi K3 demonstrating particular strength in Go programming, whereas Claude Fable 5 leads in Python, JavaScript, TypeScript, and Rust. Kimi K3's open-weight nature allows for self-hosting and flexible deployment, enhancing its appeal for teams seeking control over their AI solutions.

A new update to the inference platform offers comprehensive control over performance, cost, and quality, allowing users to deploy models quickly and efficiently without creating their own infrastructure. The platform supports multiple deployments behind a single endpoint, enabling safe updates with canary, blue-green, and rolling strategies, and testing on real traffic with A/B and shadow testing. It also introduces a closed beta for custom training, including reinforcement learning and fine-tuning, with seamless deployment to production. Open-weight models are emphasized for their control over performance, quality, and functionality, enabling enterprises to incorporate proprietary IP without exposure risks. The platform simplifies transitioning from experimentation to production, supporting open-weight, licensed closed-weight, and fine-tuned models, while offering flexibility in hardware, optimization profiles, and scalability. It features enhanced model caching for faster deployment and supports diverse autoscaling metrics. Users can maintain stable endpoints while evolving deployments, using A/B testing and shadow traffic to test changes safely. Additionally, the platform provides extensive observability tools and integrates custom training directly into the production workflow, aiming to streamline the entire model lifecycle.

Together AI and Y Combinator have partnered to create the first dedicated YC GPU cluster, aimed at providing AI-native startups within YC's portfolio easier and more cost-effective access to essential computing resources for developing and scaling AI applications. This partnership addresses the increasing challenge startups face in securing timely and affordable compute capacity, which has become a significant hurdle due to rising model quality and token value. The dedicated cluster allows startups to access GPU resources for short-term needs while benefiting from long-term pricing, enabling them to manage their own compute requirements through a self-service portal without needing to go through YC. This initiative reflects both organizations' commitment to supporting startups with the resources typically available to larger companies, ensuring that founders can focus on innovation rather than infrastructure constraints. As Together AI continues to expand its offerings and support companies as they grow, YC remains committed to funding research-driven endeavors that necessitate robust compute capabilities.

The text discusses the complexities and engineering challenges in achieving different levels of reliability in GPU inference systems, emphasizing the importance of understanding the specific failure domains covered by each reliability tier, such as node-level, data center, and regional failures. It explains that achieving 99%, 99.9%, and 99.99% reliability requires distinct architectural strategies, including automated health checks, multi-region deployments, and reserved failover capacity. The text highlights the importance of infrastructure ownership and full-stack expertise, as these factors significantly impact the ability to quickly diagnose and respond to failures. It also stresses the need for clear SLA definitions and the importance of measuring uptime at the inference completion level, rather than just at the load balancer, to ensure that the promised reliability aligns with actual performance. Furthermore, it encourages potential clients to inquire about the architecture supporting SLA claims and the provider's control over the infrastructure to ensure that the reliability promises are backed by robust systems and practices.

Recent updates to Together GPU Clusters focus on enhancing platform health and operational control to better manage large-scale training and inference workloads. The improvements include passive health checks and auto node repair to address hardware failures, alongside new operational control features like a detailed cluster view, external OIDC for Kubernetes RBAC, and startup scripts for customization. These changes aim to reduce downtime and support tickets by providing real-time detection and resolution of issues, improving cluster reliability and resilience. Additionally, Together Slurm-on-K8s 2.0 offers a revamped stack for running Slurm on Kubernetes, ensuring self-healing worker daemons, durable job accounting, and accurate GPU state management. The updates also introduce acceptance testing for larger clusters and emphasize customization and access control, providing operators with a more intuitive and efficient management experience. As Together continues to develop its platform, it seeks user feedback to further enhance its offerings.

Inkling, a multimodal mixture-of-experts model developed by Thinking Machines Lab, is designed for token-efficient reasoning, native multimodal understanding, and versatility across a broad range of tasks, including scientific reasoning, coding, and calibrated prediction. Released on Together AI's inference platform, Inkling supports text, image, and audio inputs with text outputs through a unified decoder architecture that allows developers to control the inference effort to balance reasoning depth, token usage, and latency. It features architectural innovations such as query-conditioned relative attention, short causal convolutions, and a shared expert sink within a mixture-of-experts framework, enhancing its reasoning and multimodal capabilities. The model demonstrates strong performance in preliminary evaluations, excelling across various benchmarks including scientific reasoning, mathematical problem solving, and visual document understanding. Inkling's architecture incorporates a new approach to attention mechanisms and supports scalable, efficient execution, making it suitable for diverse applications like visual question answering and multimodal agents. It is available on Together AI Serverless, providing developers with immediate access and the ability to scale from experimentation to production without the need for extensive infrastructure management.

Provisioned Throughput, a new offering from Together AI, provides reserved inference capacity for open models with token-based pricing and a 99% uptime SLA, aimed at companies seeking cost-effective AI solutions without compromising reliability. By offering a middle ground between serverless inference and dedicated inference, it allows businesses to use pre-optimized models with the predictability of guaranteed capacity, significantly reducing costs—up to 90% lower than proprietary alternatives like Claude Opus 4.8. Available for models such as MiniMax M3 and GLM-5.2, with capacity across North America and EMEA, it allows companies to optimize their inference spend according to traffic patterns without the complexity of managing infrastructure. As businesses increasingly adopt open models for diverse tasks like coding, finance, and marketing, Together AI's Provisioned Throughput provides a simplified API and pricing model, enabling them to shift from closed APIs and benefit from frontier open models' performance and cost advantages.

Together AI was founded with the vision of making generative AI widely accessible rather than dominated by a few corporations, and it has since developed a full-stack AI platform grounded in open-source principles and advanced research. The company has now secured $800 million in Series C funding from prominent investors such as Aramco Ventures, NVIDIA, and Salesforce Ventures, along with commitments for substantial compute capacity to support future growth. As AI becomes integral to production infrastructure, companies face economic challenges with proprietary models, which Together AI addresses by providing open-weight models that are cost-effective while maintaining high performance. Their platform has proven to deliver significant cost reductions, as demonstrated by companies like Decagon. The success of Together AI is driven by continuous innovation in its research-to-production pipeline, offering optimized solutions like FlashAttention-4 and Together Megakernel, which enhance the performance and economics of AI production. The company emphasizes the importance of the entire AI stack, including models, kernels, and training infrastructure, to provide comprehensive generative AI capabilities, and it continues to expand its influence in the global open-source AI ecosystem while actively recruiting talent to support its mission.