DeepInfer: DeepSeek's Open-Source Inference Framework

DeepInfer is DeepSeek’s Apache 2.0-licensed inference framework, released on October 20, 2026, for serving DeepSeek-V4.1-Flash with optimized kernels, parallelism, speculative decoding, asynchronous scheduling, structured outputs, and optional batch-invariant deterministic inference. Its deterministic NVIDIA mode addresses output variation caused by changing GPU batch composition, but requires batch invariance enabled, variable-length speculative verification disabled, and a fixed request seed, trading some throughput for reproducible CI and evaluation results. The initial release supports only NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs on Linux, targets large-scale deployments rather than consumer hardware, and excludes production infrastructure such as distributed KV-cache storage, prefill/decode disaggregation, and load balancing. DeepInfer includes golden perplexity baselines, structured-output and long-context checks, and an evaluation harness that separates generation from scoring and compares LLM judges with deterministic rule-based judges where possible. Its companion deepseek-recipe library reproduces DeepSeek API request handling while protecting against typed special-token strings being interpreted as control tokens. The material recommends using deterministic configurations and reviewed baselines for regression testing, while separately measuring agent behavior, scoring consistency, and self-hosted deployment differences.

Use this post, company, and trend context to find content marketing opportunities, perform competitive analysis, or address product feature gaps via the Plushcap MCP server or the Plushcap API.