DeepSeek open-sources DeepInfer, its production inference engine Other

DeepSeek has open-sourced DeepInfer, the inference framework that runs its own production services. The code is on GitHub under the Apache 2.0 license. With it, DeepSeek also released deepseek-recipe under the MIT license: the layer that turns API requests into model prompts and turns raw model output back into API responses.

The open-source release covers offline inference. DeepSeek says a full service still needs a KV cache system, prefill/decode disaggregation and load balancing across replicas, which are not included.

What is inside DeepInfer

• Custom kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels.
• Parallelism for MoE models: attention data parallelism, balanced context parallelism and expert parallelism with load balancing (EPLB).
• Async scheduler: prepares the next batch on the CPU while the GPU computes the current one. On by default.
• Speculative decoding: the model drafts several tokens ahead and checks them in parallel, keeping the ones that pass. This makes generation faster and cheaper.
• Determinism: batch-invariant inference and deterministic sampling. The same input gives the same output no matter the load or batch, which helps with debugging and evaluations. NVIDIA only.
• Structured outputs: limits generation to a JSON schema, a regular expression or an EBNF grammar.

What it runs on

deepseek-recipe

Self-hosted open models often break at the prompt format step: a wrong chat template or a badly parsed tool call. deepseek-recipe handles that step for DeepSeek V4 and V4.1.

• Renders conversations into prompts and tokenizes them.
• Parses raw output back into Chat Completions, Responses or Messages format, including reasoning, tool calls, JSON output and stop sequences.
• Written in Rust, with Python bindings (pip install deepseek-recipe-python).
• You still bring the inference engine, tool execution and HTTP server.

Sources

• GitHub: deepseek-ai/DeepInfer
• GitHub: deepseek-ai/deepseek-recipe