November 2026 Summaries

1 posts from Inference

Filter
Month: Year:
Post Summaries Back to Blog
A company describes building ChatTokenizer, a Rust-and-Elixir library that tokenizes inference requests in its Elixir router and sends token IDs directly to vLLM or SGLang, avoiding repeated preprocessing in their Python frontends. In internal benchmarks, this approach increased long-prompt throughput from 31 to 1,031 requests per second for SGLang and from 43 to 505 for vLLM, while ChatTokenizer’s preprocessing was reported as 53–181 times faster than the engines’ own. The system also matched vLLM’s token IDs exactly across the tested requests, an important requirement because chat templates and engine-specific rules can change the prompt the model receives. Performance improvements came from optimizations including native chat-template rendering, reduced JSON parsing, caching repeated conversation segments, and reading image dimensions from encoded image headers without decoding image data; for requests with roughly one million tokens and eight images, the authors report reducing single-request processing time from 240 milliseconds to 28 milliseconds. The post also notes that caching helped agentic conversations, while parsing and template rendering remain bottlenecks, and that the results come from controlled benchmarks on an eight-vCPU machine.
Nov 20, 2026 2,938 words in the original blog post.