DeepSeek-V4.1-Flash and the 890-byte KV cache

This paper reads DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression by DeepSeek-AI, submitted on 17 September 2026 under an author list of more than 590 names. The work and every figure, table and quotation below are theirs. What is ours is the reading: which number matters, what it is a number of, and what the compression cost. We have not trained, served or reproduced this model.

Attention compute stopped being the long-context bottleneck a while ago. Sparsity handled it. What is left is storage. Every token in flight holds bytes in HBM for the life of the request, and every cached prefix holds bytes on SSD for as long as you keep it. DeepSeek's position is that this, not FLOPs, now sets the price of a long-horizon agent: large KV caches "continue to strain HBM and SSD capacity and data-transfer bandwidth."

DeepSeek-V4.1-Flash brings the global KV cache to 890 bytes per token.

1. The number, and what it is a number of

The 890 bytes is the global KV cache, the part that is always resident in HBM, per token and independent of sequence length. At equal sequence lengths it is "roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash." Separately, the persistent cache on SSD or in host memory falls to "roughly 1/8" of V4-Flash.

Read the baseline carefully, because the headline invites a misreading. The 4x is measured against a model that was already compressed: V4-Flash runs a sparse global branch with sequence-dimension compression plus sliding-window attention. This is a fourth reduction on top of that. The cumulative figure is in Figure 1(b): "approximately 4-fold and 437-fold reductions in per-token global KV cache size relative to DeepSeek-V4-Flash and DeepSeek-V1, respectively." By our arithmetic that puts V1 near 389 kilobytes per token.

The model is 552B backbone parameters plus 196B Engram parameters, 40 layers, hidden dimension 5120, trained on 45T multimodal tokens, activating 8B parameters per token in prefill and 16B in decode.

2. The mechanism: forty layers, four caches

Compressed Sparse Attention 2 attacks three multiplicative dimensions at once, and the paper is explicit that this is the contribution: entry size, sequence position, and layer. Prior work took one each. IndexCache reuses Top-K indices across layers but "index reuse alone saves no main KV storage." Cross-layer attention shares the cache but not the selection. CSA2 decouples the two. Every layer is statically assigned one of three modes:

• Full Mode computes its own main KV, projects indexer K from that main KV, and produces fresh Top-512 indices.
• Reindex Mode reuses a preceding layer's main KV and indexer K, but computes its own indexer Q, rescores the shared keys, and selects its own Top-512.
• Reuse Mode inherits both the cache and the indices, and runs sparse attention with no indexer work at all.

All three compute their own query and their own layer-local sliding-window KV.

Now count the published assignment. The first two layers are pure sliding window. The 18 remaining encoder layers run CSA2 at compression ratio 2, in three groups of six, each one Full Mode layer and five Reuse. The 20 decoder layers run at ratio 1, in five groups of four: the first group one Full and three Reuse, the other four one Reindex and three Reuse.

So the entire 40-layer model is served by four global KV caches: three in the encoder at 2:1 sequence compression, one in the decoder uncompressed. Thirty of the forty layers do no indexing at all, which is why those layers "execute with only 15 kernels during prefill and 11 during decode." That ratio is where the compression comes from. FP4 is the smaller half of the story.

3. Reconstructing the 890 bytes

The paper gives the number but not the arithmetic. Ours, from the published configuration, lands close enough to be worth showing, because it says which component to attack next.

Main KV is a 512-channel latent. The format is "E2M1 with one E4M3 scale per 16 channels": 0.5 bytes per channel plus 32 one-byte scales, so 288 bytes per entry. Indexer K has head dimension 128, so 72 bytes per entry on the same format.

Against a published 890, that is within about one per cent, and it says the footprint is 80 per cent main KV. It also says the layer dimension is nearly spent. You cannot go below one cache. The next factor of two has to come from entry size or from a sub-four-bit format, and at four bits the paper is already trading accuracy for portability: it picks MXFP4 for the indexer "despite the higher accuracy of alternative formats in our experiments."

The dropped NVFP4 global scale is justified by bounding the magnitude rather than measuring the loss, which is unusually clean: largest trained RMSNorm weight near 1, the L2 norm of a 512-channel latent at most around the square root of 512, so roughly 22.6 after RoPE, against a format range of 2688. Observed maximum during training was "around 10." The second-level scale was carrying nothing.

4. CED, and why prefill is now the cheaper half

The Causal Encoder-Decoder is the reason the model activates 8B parameters in prefill and 16B in decode. Following YOCO, the bottom 20 layers are treated as a causal encoder; for the upper 20, global KV entries "are not derived from their respective hidden states" but projected directly from the final encoder hidden state with layer-dependent weights. Prefill therefore runs half the network: complexity falls from O(NL) to approximately O(NL/2).

The interesting part is what breaks. Sliding-window KV is still generated per layer from that layer's own hidden state, so a decoder layer's SWA KV cannot be projected from the encoder. Exact reconstruction needs a forward pass over the last n_win times L/2 tokens: at a window of 128 and 20 decoder layers, 2,560 of them. On short multi-turn prompts the paper calls that overhead "non-negligible."

Their answer is to stop reconstructing it exactly. Decoder SWA Bounded Replay replays only the last 128 tokens and truncates the window to the replay segment. The resulting state "is not mathematically equivalent to that from a full decoder forward pass." They simulate the same replay during post-training so the model adapts to it.

5. The best idea in the paper is a retention policy

The persistent cache going to 1/8 is two multiplicative factors: the global KV in it is 4x smaller, and SWA KV is no longer in it at all. The second is the one worth copying, and it is not an architecture change.

The reasoning is a cache-lifetime argument, stated plainly: "Unlike global KV, which exhibits long-tail reuse, SWA KV is reused only within a narrow, minute-scale window inside an active session and becomes dead once the session ends or the next turn begins." V4 had been holding it in an SSD tier provisioned for 72-hour residency, where it accounted for "nearly half of the persistent KV cache capacity."

V4.1 moves it to a distributed pool carved from 10 per cent of each machine's host DRAM, TTL in minutes. Far less aggregate capacity, far higher turnover. The eviction that follows is survivable only because Encoder SWA Bounded Replay turns a miss from an L times n_win forward pass into an n_win one. It "turns a catastrophic miss into a graceful, inexpensive degradation, thereby justifying the removal of SWA KV."

Any team running a prefix cache can apply that without touching a model. Match the store's retention to the data's reuse window, then make the miss cheap enough that you can afford to be wrong. It is the least novel-sounding thing here and the most portable.

6. What it cost, and what is not measured

Here is the problem with a paper titled "Pushing the Limits of KV Cache Compression." The limits are never located, because the quality cost of each step is never given a number.

FP4 main KV caching: "only marginal performance degradation." SWA Bounded Replay: "only negligible performance degradation." Encoder replay: "barely compromises response quality." Single-Pass mHC: "negligible performance degradation." Four claims, four adjectives, no ablation table. No experiment in this report isolates CSA2 mode assignment, FP4 against FP8, or bounded replay against exact reconstruction.

What the base-model table does show, V4-Flash against V4.1-Flash, is not uniform improvement:

MGSM falls 5.5 points below the model this one replaces. LongBench-V2, the only long-context benchmark in the table, moves 0.5 points and sits 6.3 behind V4-Pro. For a report whose subject is million-token serving, that is thin long-context evidence, and nothing in it separates a data effect from a compression effect: V4.1-Flash is a different size, on a different corpus, with a different data pipeline. "Despite its much smaller KV cache footprint, the model delivers substantially better performance" is true in aggregate and unattributable in detail.

Post-training results are genuinely strong, and should be read with the scaffold spread in view. DeepSWE v1.1 at 74.2 is the mini-SWE harness; the same checkpoint ranges from 65.5 to 74.2 across eight configurations, and reads 69.8 under Claude Code. Terminal-Bench 4.0 reaches 31.2 against Opus-5 at 51.8, a gap the authors name themselves.

7. Our position: this is a reproducibility change, not only a cost change

The authors put their caveat in the limitations section. We would put it in the model card.

Count the approximations now stacked in the serving path: FP4 quantisation of the cache, Top-512 selection standing in for full attention, index reuse across up to five following layers, and truncated-window replay for encoder and decoder SWA alike. None is mathematically equivalent to the exact computation. The paper is candid about where that lands:

the global KV and SWA KV computed for the uncached suffix depend on the cache-hit position and are not mathematically identical across positions

Read that as an API contract. The output of this model is a function of the prompt and of the cluster's cache state when the request arrives. Two identical requests, one landing on a warm prefix and one on a cold one, run measurably different computations, and at temperature zero they can still differ.

That is our position and it is arguable: for anyone building evaluation harnesses, regression suites or agent scaffolds on a served endpoint, this matters more than the cost saving. A flaky agent run is no longer evidence of a flaky agent. The authors list "SWA state reconstruction at cache-resumption boundaries" as future stress-testing work, conceding the point without drawing the conclusion. Ours is that cache-state-dependent output belongs in the endpoint's documented behaviour, not in a limitations paragraph.

A second, smaller position. The paper treats the layer dimension as open frontier. Our reconstruction says it is nearly closed: four caches, three already at 2:1, one irreducible. The next 4x comes from entry size instead, the axis MLA opened in DeepSeek-V2 and the one Kimi K3 spent its Gated MLA layers on. It has been quiet for two years and is about to get loud again. The sibling axis, parameter sparsity, is the one we covered in mixture of experts: what actually changed, and this model extends its auxiliary-loss-free balancing with separate per-modality biases.

8. What it means for serving

Our arithmetic, not theirs. At 890 bytes per token a full one-million-token context holds about 890 MB of global KV. On one 80 GB device, ignoring weights and SWA state, that is roughly 90 concurrent maximum-length contexts against roughly 22 for V4-Flash. Concurrency is what sets price per token on a long-context endpoint, so a 4x cache reduction is close to a 4x reduction in the storage-bound floor on serving cost. That tier is already the cheapest end of what the frontier costs.

One caution on the compute claim. Figure 2 reports decode FLOPs "weighting BF16, FP8, and FP4 operations by 1, 0.5, and 0.25," and shows a 256-fold context increase costing only a quarter more. That is a modelled quantity with the precision weighting baked in. There is no tokens-per-second figure, no latency distribution and no cost-per-million in the report, so the cost case is an architecture argument, not a benchmark.

One last observation. DeepSeek-ViT adopts RMSNorm, SwiGLU and 2D-RoPE, exactly the language-to-vision transplant set that ViT-5 swept component by component and found partly harmful. No ablation is offered for it here either.

9. Sources

1. DeepSeek-AI, DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression, arXiv:2609.19969, 17 September 2026. Every number, table row and quotation above is from this report unless labelled as ours.
2. DeepSeek-AI, DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence, the baseline every compression ratio in this paper is measured against.
3. DeepSeek-AI, DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model, where Multi-head Latent Attention is introduced and where the abstract claims a 93.3 per cent KV cache reduction against DeepSeek 67B.
4. Sun et al., You Only Cache Once: Decoder-Decoder Architectures for Language Models, the YOCO design CED is explicitly built on.
5. Brandon et al., Reducing Transformer Key-Value Cache Size with Cross-Layer Attention, the cross-layer sharing precedent, which reports a further 2x against MQA.
6. Chen et al., PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention, cited in the paper as the evidence that SWA's effective receptive field is far smaller than its theoretical one, which is what licenses bounded replay.
7. Rouhani et al., Microscaling Data Formats for Deep Learning, the OCP microscaling formats behind the FP4 indexer cache.
8. Our reading of Mixture of experts: what actually changed, which covers DeepSeek-V3 and the auxiliary-loss-free load balancing that this model extends with separate per-modality correction biases.
9. Our reading of Kimi K3: where the compute went, for the comparison case of a model that kept periodic dense global attention instead.

We have not trained, served, quantised or reproduced any part of DeepSeek-V4.1-Flash, and we have not run any of the benchmarks named above. The byte reconstruction in section 3 is our arithmetic from the published configuration and rests on one assumption the paper does not state, that indexer K is stored as a single 128-dimensional head per entry in the same FP4 format as main KV; it lands within one per cent of their figure, which is suggestive and not proof. The positions in sections 5, 7 and 8 are ours and the authors do not state them. We could not verify the serving-cost arithmetic in section 8 against any published throughput number, because the report contains none.

This note sits in our Engineering track, alongside the agentic and foundation models threads. The full library is at Research.