The kernel stack, assembled
FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels are wired into one engine. DeepSeek had released the pieces before; this release wires them into one engine.
FIG. 01 — Release dossier / 2026.10
DeepInfer, read closely.
DeepInfer is DeepSeek's open-source inference framework for serving DeepSeek-V4.1-Flash at scale. Released on 20 October 2026 under the Apache-2.0 license, it provides an offline inference engine for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs — but not the KV cache system, prefill/decode disaggregation, or load balancing needed for a complete online service.
Released20 Oct 2026UTC
Code licenseApache-2.0weights: MIT
ModelV4.1-Flashonly
HardwareSM100 / Ascend 950Linux only
ScopeOffline enginenot a hosted API
The short version
An engine, a boundary, and a set of numbers that need asterisks.
Read time: 6 minutes
What it is
A production-derived inference engine: optimized kernels, expert parallelism, speculative decoding, sliding-window cache recovery, asynchronous scheduling, and deterministic modes. DeepSeek also released deepseek-recipe, a companion library for API protocol conversion and tokenization.
What it isn't
- Not a model. The model is V4.1-Flash.
- Not a hosted API or an OpenAI-compatible server.
- Not a complete serving stack: KV cache, PD disaggregation, and load balancing are missing.
- Not a general-purpose engine for other models.
Why it matters
It is a rare look at how DeepSeek actually serves a frontier open-weight model: Wide-EP at decode, confidence-scheduled speculation, and a cache policy that trades exactness for bounded recompute. The ideas are useful even if you never run the code.
The boundary, in one sentence. DeepInfer is an offline inference engine, not a hosted API or a model; the release does not include the KV cache system, prefill/decode disaggregation, or data-parallel load balancing needed for a complete online service.
Scope
What actually shipped
Official README + Q&A
Wide-EP and EPLB
Attention uses data parallelism; the MoE part uses expert parallelism. In DeepSeek's production design, decode spreads experts across 64–128 GPUs and rebalances hot experts with redundant copies.
DSpark, with variable verification
Draft tokens are proposed ahead; a confidence head predicts which ones are worth verifying. DeepSeek reports +4–7% throughput at its production operating point.
SWA bounded replay
Instead of rebuilding the full sliding-window cache across every layer, the engine replays only the recent tail. The result is approximate; the recompute cost becomes bounded.
Async by default
The CPU prepares the next batch while the GPU runs the current one. The goal is to remove the idle gap between decode steps — an engineering detail with real throughput value.
Deterministic modes
Batch-invariant inference and deterministic sampling are available on NVIDIA. They are the unglamorous features that make benchmarks and agent runs auditable.
Included
- Offline inference for DeepSeek-V4.1-Flash
- Optimized kernels and model / sampler runtime
- Attention DP, balanced CP, MoE EP + EPLB
- DSpark, SWA bounded replay, async scheduling
- Batch-invariant and deterministic modes (NVIDIA)
- Structured outputs: regex, JSON Schema, EBNF
- deepseek-recipe (separate MIT repository)
Not included
- A KV cache system
- VE / Prefill / Decode disaggregation
- Load balancing across data-parallel replicas
- A user-facing online API
- Support for other DeepSeek models
- Published Ascend 950 performance data
- Third-party reproduction of the charts
Numbers
The benchmarks, with their asterisks
Official charts / 131,072 context
131,072
Context length in both charts
Not the model's 1M-token maximum.
4 + 64
SM100 GPUs used for prefill + decode
Each within one scale-up domain.
0
Third-party reproductions we could find
As of 2026-10-20 11:30 UTC.
Read the axes, not the headline. Prefill is throughput vs per-step latency. Decode is throughput vs per-user speed. Neither is a promise about end-to-end request latency, cost per token, or your workload. The decode chart also assumes a 75% draft-acceptance rate; harder text lowers the gain.
The gap
The part most launch posts skipped
Three missing pieces
DeepInfer is an engine, not a service. The README is unusually direct about the difference: a complete inference service also needs a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. None of the three is in the release.
The gap is not a scandal. It is a design boundary: DeepSeek kept the parts most tightly coupled to its internal infrastructure. But it changes who can use the code and what they can expect from it.
What you would have to build
- A prefix/KV cache that survives across requests and machines
- Separate prefill and decode pools with transfer and routing
- Load balancing that is aware of both request count and cache length
- An API server, auth, quotas, billing, and observability
Name check
Which DeepInfer are you reading?
One name, several projects
DeepSeek2026 · LLM inference · this site
Medical2016 · 3D Slicer · unrelated
Smart contracts2023 · bytecode types · unrelated
DeepInfradifferent spelling · different company
The name DeepInfer also refers to an unrelated 2016 medical-imaging toolkit and an unrelated 2023 smart-contract type-inference project; this site covers only DeepSeek's 2026 inference framework. Full disambiguation →
Launch notebook
How we got here
Times in UTC
- 2025-04
DeepSeek says a full inference-engine release is impractical: the code diverged from vLLM, is coupled to internal infrastructure, and the team lacks maintenance bandwidth.
- 2026-09-30
Ascend groundwork: DeepGEMM-Ascend, DeepEP-Ascend, and Ascend backends for the kernel stack. No DeepInfer yet.
- 2026-10-16
The
deepseek-ai/DeepInferrepository is created. - 2026-10-20 09:30
Official X thread and public repository. The code lands with three public commits.
- 2026-10-20 11:26
Snapshot: about 5.4k stars, 408 forks, 47 open items. A 120-comment Hacker News thread is already running.
Community signal
A recurring technical objection was not about speed. It was about the missing KV cache. Within hours, contributors also opened RFCs for SM90/H20 and Ascend 910C support — demand beyond the official hardware list, not shipped support.
What we did not do
We did not run DeepInfer. We do not have SM100 or Ascend 950 hardware. Every number on this site is attributed to a public source, and every claim we could not verify is labeled as such.
Read next
Go deeper
Six entries
What shipped
Scope, build requirements, licenses, and the repository map.
/release/ →How it works
Wide-EP, DSpark, SWA replay, async scheduling, and Recipe.
/architecture/ →Benchmarks
The two charts, the eval numbers, and what they don't prove.
/benchmarks/ →Limits
Who can run it, what's missing, and what you'd have to build.
/limits/ →Name
DeepSeek vs medical vs smart contracts vs DeepInfra.
/name/ →FAQ
Sixteen questions, each answered with its source.
/faq/ →