Content Deep Dive

Launching a trillion-parameter model at Sakana AI

Blog post from Modal

Post Details
Company
Date Published
Author
-
Word Count
1,280
Company Posts That Month
17
Language
English
Hacker News Points
-
Post removed?
No
Summary

Modal and Sakana AI collaborated to deploy Sakana Namazu, an approximately one-trillion-parameter mixture-of-experts model designed for Japanese business contexts, web search, and code execution, using dynamically scalable GPU infrastructure rather than reserved capacity. To reduce memory demands while preserving Japanese-language performance, they applied NVFP4 quantization selectively to routed experts while retaining BF16 precision for attention, routing, and shared experts, calibrating the model with representative Japanese-style conversations. They also developed custom speculative decoding draft models trained on more than 30 billion raw production-style token IDs, later refining the approach with data from roughly one million real requests; Modal reports that this increased token acceptance, reduced decoding latency, and improved p99 latency. Additional serving optimizations addressed CPU–GPU synchronization, expert-matrix kernels, cached setup work, multi-GPU acceptance consistency, and grammar-constrained structured outputs. After launch, a KV-cache-aware router reportedly reduced queueing and time-to-first-token spikes by directing requests toward replicas likely to contain cached prompt prefixes, while elastic GPU scaling helped accommodate uncertain launch demand.

Trends Found in this Post
Trend Post Mentions Total Month Mentions Posts Companies MoM
Serverless 1 1,385 326 109 +13%
Use This Data

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.