DeepSeek-V4.1-Flash runs 5.3x faster on vLLM after three weeks of optimization

The vLLM community and Inferact pushed DeepSeek-V4.1-Flash to a 1.9× improvement in time-to-first-token at low concurrency and a 5.3× throughput gain under a 150-tokens-per-second-per-user constraint on SemiAnalysis’s AgentX benchmark — all within three weeks of release. The numbers come from the developers’ own tests, not an independent lab, so they reflect a theoretical ceiling more than a typical production environment.

The architecture behind the jump

DeepSeek V4.1 introduces a causal encoder-decoder (CED) architecture that activates 16 billion parameters per token during decode but only 8 billion during prefill. The model is also radically memory-efficient: it combines Compressed Sparse Attention 2 (CSA2), an FP4 KV cache, and cross-layer KV sharing to drive the global footprint down to 890 bytes per token. By comparison, comparable FP16 models sit at 2–3 kilobytes per token — a gap that translates directly into the ability to hold longer context on the same hardware.

SWA bounded replay, or how to skip half the model

The model maintains two KV caches: a global cache (compressed, layer-shared, FP4, ~890 bytes/token) and a sliding-window attention (SWA) cache (uncompressed FP8, covering the most recent 128 positions across all 40 layers). The SWA cache imposes two costs: prefix-caching at every possible boundary costs over 10× more than the global cache, and prefill runs layers 21–39 on every prompt token even though decode only reads the last 128. DeepSeek’s solution, SWA bounded replay, re-runs only those last 128 tokens and truncates the window at the start of replay — not bit-exact, but the company says quality loss is negligible. vLLM implements this in two places: on the encoder side, it keeps only the global KV and replays the SWA on a prefix hit; on the decoder side, it runs layer 20 on every token to produce the global KV, then runs layers 21–39 only on the last 128 tokens of each request, skipping nearly half the model on long prompts.

CUDA graphs and the new kernels

After the pruning, layers 21–39 do so little work on the GPU that kernel-launch overhead dominates and the GPU sits idle. Their input shapes also differ from layers 0–20, so they can’t be captured in a single CUDA graph. vLLM’s existing PIECEWISE mechanism already knows how to split mid-model, so it captures 0–20 on the full batch and 21–39 separately on the pruned batch, enabling better graph coverage for the pruned section. At the same time, DeepSeek’s new kernels — MegaAttention, Mega-mHC, Mega-Gate, and DeepSelect — were merged, and the remaining ops were aggressively fused and parallelized, including mHC side streams and all-reduce fused with the operators before and after into a single kernel.

What this means in practice

The combination of model-level optimizations (CED, CSA2, FP4 KV, cross-layer sharing) and system-level optimizations (SWA bounded replay, split graphs, kernel fusion) yielded the jump from 1.9× on TTFT to 5.3× on throughput under load. DeepSeek has not published quantitative quality metrics for the SWA bounded replay loss beyond “negligible,” and there is still no independent run on AgentX confirming the numbers in an uncontrolled environment. Still, it’s a clean demonstration of how an architecture designed from the ground up for long-horizon serving lets the inference system cut corners without breaking the model — a pattern we’ll likely see more of in the next generation of agent models.