The Inference Stack Behind OCRia

OCRia-1’s document-parsing inference speed was improved from roughly 285 to 2,800 tokens per second on a single NVIDIA B200 GPU by targeting the memory-bound decoding phase that dominates latency for long structured document outputs. The work combined workload-aware mixed-precision quantization, which retained higher precision for sensitive components such as the vision encoder, with speculative decoding using DFlash2 to draft and verify multiple tokens per target-model pass without changing the target model’s output distribution. Further gains came from optimizing recurrent Gated Delta Network state handling with ReplaySSM, replacing slower Triton kernels with custom CUDA implementations, fusing GPU operations to avoid unnecessary high-bandwidth-memory transfers, and reducing repeated CPU-side metadata preparation in vLLM. Although many additional kernel and engine-level refinements individually contributed modest improvements, together they added more than 60% throughput improvement while evaluations were used to preserve task accuracy and numerical equivalence. The optimized OCRia-1 model is available through LlamaParse, while broadly applicable changes are intended for contribution to the vLLM community.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.