Together Compiler: Continual Compilation for AI inference

Together Compiler (TKCC) is an agent-driven continual optimization system designed to adapt AI inference engines to rapidly changing combinations of models, hardware, traffic patterns, and user workloads. Together AI argues that manual tuning and fixed compiler heuristics cannot scale across diverse architectures, including mixture-of-experts and sparse-attention models, or across hardware platforms with different performance characteristics. TKCC accelerates experimentation through persistent inference engines that hot-reload candidate optimizations instead of repeatedly restarting, multiple collaborating agents, and a two-tier measurement system that balances rapid testing with validation on realistic data distributions. It also uses a cost model combining hardware information, microbenchmarks, and measured forward passes to choose parallelism configurations as traffic shifts. The company reports that TKCC achieves up to 2.6 times higher throughput than optimized open-source inference baselines, reaches reported low-latency benchmark results of 1,111 tokens per second for MiniMax M3.1-Flash and 1,205 for DeepSeek v4.1-Flash, and outperforms open-source megakernels by 1.7 times on a Llama 1B workload. Beyond language-model serving, the system is presented as applicable to weather modeling, partial differential equations, time-series analysis, and biology workloads, reflecting Together AI’s broader concept of “polymorphic” systems that continuously reoptimize as conditions evolve.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.