DeepSeek has released the engine it uses to serve its models in production. DeepInfer, licensed under Apache 2.0, hit Hacker News on October 20, 2026 and within hours had 565 points and 201 comments on Hacker News. It is not a new vLLM for everyone: today it runs exactly one model, on two kinds of hardware, in offline mode. But it shows from the inside how a Chinese lab serves an open-weights MoE at scale.
What was released
According to its README, DeepInfer is the high-performance inference framework behind DeepSeek's production services. This first drop covers offline inference with the core pieces needed to serve at scale:
| Area | What it supports |
|---|---|
| Models | DeepSeek-V4.1-Flash (checkpoint DeepSeek-V4.1-Flash-1017) |
| Hardware | NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs |
| Parallelism | Attention DP, balanced context parallel, expert parallel with EPLB |
| Decoding | DSpark: draft tokens with confidence-scheduled verification |
| Reproducibility | Batch-invariant inference and deterministic sampling (NVIDIA only) |
| Structured output | Regex, JSON Schema or EBNF grammar |
| Requirements | Linux, GCC 13+, Python 3.12, CUDA 13 or CANN 9.2 |
The kernels are the ones DeepSeek had already open-sourced one by one (FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect), plus custom TileLang kernels. The repo's performance runs used 4 GPUs for prefill and 64 for decode, each group within a single scale-up domain.
What is missing
The README is upfront about it: a complete service needs three more pieces. There is no KV cache system, prefill/decode disaggregation into separate services is not included (although most of the engine side is already implemented), and there is no load balancing across data-parallel replicas. In other words, they released the engine but not the platform around it, which is exactly where much of the work of running an inference service lives.
How it compares with vLLM
By its authors' own account, DeepInfer was "originally built primarily on vLLM", with ideas from SGLang and TensorRT-LLM. The comparison is unavoidable, and it makes clear what kind of project each one is:
| DeepInfer | vLLM | |
|---|---|---|
| Models | 1 (DeepSeek-V4.1-Flash) | 200+ Hugging Face architectures |
| Hardware | NVIDIA SM100, Ascend 950 | NVIDIA, AMD and Intel GPUs, CPUs, plugins for TPU, Ascend, Gaudi and more |
| API | Offline batch | OpenAI-compatible server, Anthropic Messages and gRPC |
| License | Apache 2.0 | Apache 2.0 |
| Focus | Squeezing one model on specific hardware | General-purpose engine |
vLLM is the tool for serving models; DeepInfer is the reference for how far a single model can be pushed when you own the whole stack. Its ideas are more likely to show up in general-purpose engines than in third-party clusters.
What it means
The most significant part is not the code itself but first-class support for Huawei Ascend 950 on par with NVIDIA. DeepSeek is publishing a production engine that runs on Huawei hardware in the same week its technology was banned across US federal civilian agencies. For anyone buying inference capacity, it is a sign that the non-CUDA ecosystem now has a real at-scale serving story.
For people running open models, two ideas are worth tracking. The first is batch-invariant inference: getting the same result no matter which other requests share your batch is what lets you reproduce a bug or evaluate a model without noise, and DeepInfer exposes it behind a single environment variable. The second is DSpark, speculative decoding where draft verification is scheduled by the model's confidence instead of always checking a fixed length.
If you serve DeepSeek-V4.1-Flash in production, you now have a direct reference for how its own authors do it. For everyone else, vLLM is still the way to go.
Sources
- deepseek-ai/DeepInfer — README, hardware, features and license.
- DeepSeek just open-sourced its inference engine — Hacker News, October 20, 2026.
- vllm-project/vllm — supported models, hardware and APIs.
- US bans DeepSeek technology across federal civilian agencies — The Decoder, October 20, 2026.