How to serve trillions of tokens for trillion-parameter coding agents
Blog post from Modal
Modal describes how it optimized large-scale inference for coding agents using Moonshot AI’s trillion-parameter Kimi K2.6 model, raising per-replica performance by 2.8 times per user and 5.6 times across users while supporting services that processed hundreds of billions of tokens daily. The company explains that coding-agent workloads are dominated by long, highly overlapping session histories, making key-value cache management, low latency, and high token throughput central challenges. Its approach combined tensor parallelism across four GPUs, customized speculative decoding with a fine-tuned DFlash draft model, memory reductions through FP8 cache quantization and NVFP4 shared experts, and SGLang’s multi-tier HiCache system extending cache storage into CPU memory. Modal also found that scaling required cache-aware, load-aware routing rather than simple session-affinity hashing, because uneven session sizes, concurrent requests within sessions, and cache relocations during scale-ups created overloaded replicas and tail latency. The post emphasizes that inference performance must be evaluated against representative workloads, since output length, cache behavior, and speculative-decoding acceptance rates vary substantially by data, and notes that the resulting methods have been applied to newer models and released through SGLang contributions and Modal endpoint configurations.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| LLM | 3 | 6,051 | 1,178 | 246 | +14% |
| AI Model Fine-tuning | 2 | 741 | 184 | 68 | +29% |
| Real-time | 2 | 4,868 | 1,094 | 243 | +4% |
| Harness engineering | 1 | 164 | 114 | 62 | -22% |
| Kubernetes | 1 | 4,372 | 413 | 104 | +23% |
| Local AI | 1 | 215 | 68 | 21 | -18% |
| Observability | 1 | 2,783 | 740 | 189 | -15% |
| Serverless | 1 | 1,231 | 285 | 104 | +38% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.