We made vLLM tokenization 120x faster with Elixir
Blog post from Inference
A company describes building ChatTokenizer, a Rust-and-Elixir library that tokenizes inference requests in its Elixir router and sends token IDs directly to vLLM or SGLang, avoiding repeated preprocessing in their Python frontends. In internal benchmarks, this approach increased long-prompt throughput from 31 to 1,031 requests per second for SGLang and from 43 to 505 for vLLM, while ChatTokenizer’s preprocessing was reported as 53–181 times faster than the engines’ own. The system also matched vLLM’s token IDs exactly across the tested requests, an important requirement because chat templates and engine-specific rules can change the prompt the model receives. Performance improvements came from optimizations including native chat-template rendering, reduced JSON parsing, caching repeated conversation segments, and reading image dimensions from encoded image headers without decoding image data; for requests with roughly one million tokens and eight images, the authors report reducing single-request processing time from 240 milliseconds to 28 milliseconds. The post also notes that caching helped agentic conversations, while parsing and template rendering remain bottlenecks, and that the results come from controlled benchmarks on an eight-vCPU machine.
| Trend | Post Mentions | Total Month Mentions | Posts | Companies | MoM |
|---|---|---|---|---|---|
| LLM | 1 | 3,674 | 823 | 199 | -44% |
Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.