DeepInfer Limits: KV Cache, PD, and Who Can Run It

DeepInfer is an offline inference engine, not a hosted API or a model; the release does not include the KV cache system, prefill/decode disaggregation, or data-parallel load balancing needed for a complete online service. It serves DeepSeek-V4.1-Flash, and that gap is the README's own list.

The three missing pieces

1. A KV cache system

Every request builds a KV cache: the keys and values the model has already computed for the prompt. If the same prefix appears again — a long system prompt, a multi-turn agent history, a shared document — reusing the cache avoids recomputing it. At scale, prefix-cache hit rate is one of the biggest levers on cost and time-to-first-token.

A real KV cache system is not a dictionary in GPU memory. It needs:

• a storage hierarchy (HBM, host memory, SSD) with eviction and TTL;
• a way to find a prefix across machines, not just locally;
• transfer paths that overlap with computation;
• fault tolerance, because compute nodes fail more often than storage nodes;
• a routing policy that does not fight the load balancer.

DeepSeek's production design gives a sense of the scale: a dedicated 3FS cluster with roughly 100 PiB of SSD, 10+2 erasure coding, a 3-day TTL for agent traffic and 7 days for chatbot traffic, about 14.4 TB of logical KV per GPU, and roughly 400 Gbps of interconnect per 8 GPUs. None of that is in the release.

2. VE / Prefill / Decode disaggregation

The release says “most required engine functionality is already implemented,” but the full disaggregation scheme is not included. In production, the vision encoder, prefill pool, and decode pool run on separate machines with separate scaling policies. The hard parts are the handoff: moving KV cache between pools, routing requests to the right prefill engine, and keeping decode occupancy balanced.

Without this, you can still run offline batches. You cannot reproduce DeepSeek's online service shape.

3. Load balancing across data-parallel replicas

DeepSeek uses separate balancers for prefill, decode, vision encoder, and expert parallelism. The prefill balancer buckets requests by context length and routes by remaining compute quota; the decode balancer considers both request count and total KV length; the expert balancer moves hot experts between GPUs. The Q&A says the result reaches roughly 80% of a perfectly balanced offline benchmark.

That last number is a useful reality check: even DeepSeek's production system reaches only about 80% of the offline benchmark's throughput. A simple round-robin router is unlikely to do better.

Why the line is where it is

In April 2025, DeepSeek explained why it was not open-sourcing the full inference engine: the code was a heavily modified fork of an older vLLM, it was coupled to internal cluster management, and the team lacked the bandwidth to maintain a large open-source project. The plan was to extract standalone libraries and contribute optimizations upstream.

DeepInfer is the next step, not a reversal. It opens the compute and scheduling path — kernels, parallelism, speculation, cache recovery, structured output — and keeps the parts most tightly coupled to internal infrastructure: the storage system, the cross-machine disaggregation, and the production load balancers.

That is a coherent boundary. It also means the “open-source production engine” framing needs a qualifier: the engine is open; the service around it is not.

Who can actually run it

DeepInfer supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, but DeepSeek's published performance charts cover NVIDIA hardware only. That does not make Ascend unsupported; it makes Ascend unmeasured in public.

What you'd have to build

If you want to turn DeepInfer into a service, budget for these:

• KV cache service: prefix matching, tiered storage, cross-machine transfer, TTL/eviction, fault tolerance.
• Prefill/decode routing: separate pools, KV handoff, capacity planning, autoscaling.
• Load balancing: prefill by compute quota and context length; decode by request count and KV length; expert rebalancing.
• API layer: Chat Completions / Responses / Messages, streaming, tool calls, structured output, error handling.
• Tenancy and operations: auth, quotas, billing, rate limits, tracing, metrics, logs, canary rollout.
• Evaluation: regression tests for quality, latency, and cost under your real workload.

DeepSeek's deepseek-recipe helps with the protocol/tokenization part of the API layer. It does not provide the server, auth, billing, or KV cache.

Alternatives, without a winner

If your goal is to serve V4.1-Flash, DeepInfer is not the only path. The right choice depends on what you already operate and what you need to control.

We are not ranking these. vLLM and SGLang already have API layers and broad adoption; DeepInfer has DeepSeek's production tuning and a narrower scope. A team that needs an API next week should not start by building a KV cache service.

Decision guide

• Use DeepInfer if you have SM100 or Ascend 950 hardware, you want to study or reproduce DeepSeek's serving optimizations, and you are prepared to build the service layer.
• Use vLLM/SGLang if you want a served API now, need broad model support, or do not want to own the KV cache and routing stack.
• Wait if you need Ascend performance data, production parity, or a supported online serving path from DeepSeek.
• Do not use it if you have consumer GPUs, need other models, or expect a turnkey OpenAI-compatible endpoint.

Next: the production gap in more detail, or the short answers.