Gated DeltaNet-2, and the gate that was doing two jobs

This paper reads Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention by Ali Hatamizadeh, Yejin Choi and Jan Kautz at NVIDIA. The work, every table and every figure below are theirs. What is ours is the reading: which result carries the claim, where the gain actually comes from, and what the fixed-size state still costs. We have not trained or reproduced any of it.

Softmax attention has no memory policy. It keeps everything. The KV cache grows with the sequence, decoding cost grows with it, and nothing is ever overwritten because nothing needs to be: the cache is the transcript.

Linear attention takes the opposite deal. Replace the cache with a fixed matrix state S in R^(d_k x d_v), update it once per token, read it with a query. Sequence mixing becomes linear in length and decoding becomes constant in memory. The cost is that memory is now finite, so writing something new means editing something old, and the interesting engineering question stops being "what do we keep" and becomes "how do we edit a compressed store without scrambling what is already in it".

Gated DeltaNet-2 is an answer to a narrow part of that question: the single gate that was controlling two unrelated decisions.

1. The delta rule, stated properly

The plainest linear attention update is pure accumulation:

S_t = S_{t-1} + k_t v_t^T
o_t = S_t^T q_t

Each token adds an outer product. Nothing is removed, so keys that repeat pile their values on top of each other and the read at that key returns the sum rather than the latest value. A state that only accumulates eventually returns mush.

The delta rule fixes it by reading before writing. Before committing v_t at key k_t, the model reads out whatever value the state currently associates with k_t, which is S_{t-1}^T k_t, and subtracts it. The paper writes the update as

S_t = (I - beta_t k_t k_t^T) S_{t-1} + beta_t k_t v_t^T

with a scalar step size beta_t in [0,1]. Read it as one gradient step on a least-squares problem: the state is a fast weight matrix being fit online to map keys to values, the residual is v_t - S_{t-1}^T k_t, and beta_t is the learning rate. At beta_t = 1 the old association at that key is fully replaced. At beta_t = 0 nothing happens. This is the fast-weight-programmer view of linear attention made operational, and the paper derives its own rule the same way, from a local objective minimised at each step.

Two refinements preceded this paper. Gated DeltaNet adds a scalar decay alpha_t in front of the state, so the whole memory can be flushed independent of any particular key:

S_t = alpha_t (I - beta_t k_t k_t^T) S_{t-1} + beta_t k_t v_t^T

Kimi Delta Attention sharpens that decay into a per-channel vector, so each key channel forgets at its own rate rather than the whole state fading together.

Note what did not change across either refinement. beta_t is still a scalar, and it still appears twice: once in beta_t k_t k_t^T, which governs how much of the old association is erased, and once in beta_t k_t v_t^T, which governs how much of the new value is committed. One number, two decisions, and no way to express "clear this slot hard but write into it tentatively".

2. What decoupling actually means here

Gated DeltaNet-2 splits the scalar into two channel-wise gates. An erase gate b_t in [0,1]^d_k on the key side, a write gate w_t in [0,1]^d_v on the value side, and from those two effective vectors:

e_t = b_t ⊙ k_t      z_t = w_t ⊙ v_t
S_t = (I - k_t e_t^T) D_t S_{t-1} + k_t z_t^T

where D_t is the channel-wise decay inherited from KDA. The asymmetry is the point. The erase term uses k_t e_t^T, not k_t k_t^T, so the direction the state is read along is no longer the direction it is written along. b_t selects which key-side coordinates of the decayed state get read out and removed. w_t selects which value-side coordinates get committed. The two can now disagree.

The family collapses cleanly, which is the cleanest thing about the formulation. Set both gates to the same scalar and you have KDA. Collapse the decay as well and you have Gated DeltaNet. The paper is a strict generalisation, not a sideways move, so a loss against the baselines would have been a genuinely informative negative result.

Making this trainable at scale is the part that would stop most people. A chunkwise WY representation keeps the update parallel: after normalising out the decay, the recurrence becomes S_r = (I - k_r e_r^T) S_{r-1} + k_r z_r^T with the channel-wise decay absorbed into what the authors call asymmetric erase factors, k_r scaled by the inverse decay and e_r scaled by the decay. Build T = tril(E K^T, -1), invert (I + T), and the same triangular solve serves both the erase-side auxiliary Y = A E and the write-side U = A Z. One solve, two right-hand sides. That amortisation is why adding a second gate costs little: Gated DeltaNet-2 keeps the near-flat scaling profile of the recurrent mixers, training throughput on an H100 dropping only from 38.0 Kt/s at 2K to 36.1 Kt/s at 16K. Note what that number is not. It is this model across sequence lengths, not this model against KDA at matched length. The authors call the gap to KDA a modest constant cost and plot it in Figure 2 without tabulating it, so the overhead of the second gate is not a figure this paper gives.

3. The numbers

1.3B parameters, 100B FineWeb-Edu tokens, 4K sequence length, 16 heads, d_k = d_v = 128, d_model = 2048. The recurrent state is 262,144 floats per layer. AdamW at peak learning rate 4e-4, batch 0.5M tokens, 1B-token warm-up.

Half a point of perplexity over Gated DeltaNet and a point of commonsense accuracy. Real, and smaller than the retrieval story.

On RULER, recurrent, the multi-key task is where the mechanism shows:

MK-NIAH-1 at 2K goes from 37.0 to 51.4 per cent, a 39 per cent relative gain, and S-NIAH-3 at 2K from 54.2 to 89.8. Multi-key retrieval is exactly the task a coupled gate should be bad at, because several distinct associations must coexist in one state without bleeding into each other, so this is the result the mechanism predicts.

Real-world retrieval, averaged over SWDE, SQuAD, FDA, TriviaQA, NQ and DROP: 29.88 per cent recurrent against 28.67 for KDA and 28.09 for Gated DeltaNet, and 42.28 hybrid against 40.14 for KDA and 38.07 for the Transformer. Roughly 1.2 points recurrent and 2.1 hybrid over KDA; against Gated DeltaNet, 1.8 and 3.2.

4. Where the gain came from

The ablation is the most useful table in the paper and the one that complicates its title. Make only one of the two gates channel-wise and leave the other scalar:

Measure both against KDA's 16.81. The full model closes 0.91 perplexity. The erase gate alone closes 0.69 of that, about 76 per cent. The write gate alone closes 0.26. On recall the split is the same shape: erase alone recovers 0.84 of the 1.21-point gain over KDA.

Our position, which the authors do not state: this is an erase-gate paper wearing a decoupling paper's title. "Decoupling erase and write" implies two findings of comparable weight. The measurement says the key-side selectivity is load-bearing and the value-side gate is a small top-up for a near-doubling of gate parameters on the value path. That matters practically, because an implementer with a tight parameter or kernel budget should take b_t and skip w_t, and the paper's framing does not make that obvious. It matters scientifically too: it suggests the real deficiency in the delta rule was never the write magnitude but the inability to erase surgically, and the sharper follow-up question is how much further key-side selectivity goes, not what symmetric gating buys.

The authors do report the asymmetry in prose, and they note that widening the erase gate from [0,1] to [0,2], which would let an edit overshoot into negative association, gives no consistent gain at this scale: 15.95 against 15.90. A clean negative worth having.

5. What the fixed state still costs

The trade is visible in the table above and the paper does not hide it, but it is worth making arithmetic rather than rhetorical.

At 4K context with these dimensions, a softmax KV cache holds roughly 4,096 floats per token against the recurrent state's fixed 262,144, so the state is about 64 times smaller at 4K and the ratio doubles with every doubling of length. Break-even is around 64 tokens. Below that the recurrent state is pure overhead; above it the saving compounds without limit. That calculation is ours, from the paper's stated dimensions.

What you pay for it is in the same tables. Recurrent S-NIAH-2 falls from 93.0 at 4K to 39.2 at 8K, on a model trained at 4K. Softmax attention does not do that, because it never compressed. The gates make the compression smarter; they do not make it stop being compression.

And one regression the abstract's phrasing walks carefully around. On recurrent S-NIAH-3 at 4K, Gated DeltaNet-2 scores 31.8 against Gated DeltaNet's 60.6, a loss of nearly half. On recurrent DROP it scores 17.87, the worst of all six recurrent models, below Mamba-2's 19.60 and KDA's 21.80. The abstract says the method "improves the evaluated multi-key retrieval setting and remains strong", which is true and is also a sentence doing some work. Our reading of the pattern: channel-wise erase sharpens commitment, and sharper commitment is worth more when the right key must be isolated from distractors and worth less when many associations must simply survive. That is a hypothesis the paper's data is consistent with and does not test.

Second position of ours. The hybrid results argue against the layer mattering as much as the recurrent results suggest. Add 2K sliding-window attention and the S-NIAH-2 @4K advantage over Gated DeltaNet collapses from 5.8 points to 0.6. Most of what the erase gate fixes, local attention also fixes, and a team choosing between a better recurrent edit rule and a small attention window should know those are substitutes before they are complements. The honest counter is that the hybrid real-world retrieval gap over Gated DeltaNet widens rather than narrows, from 1.8 points recurrent (29.88 against 28.09) to 3.2 hybrid (42.28 against 39.11), so the substitution is not total.

6. Limits

• One scale, one seed. 1.3B parameters, 100B tokens, one run per configuration. No variance bands anywhere, and the margins in Table 5 are of a size that seed noise could plausibly produce. The gate-structure ordering is consistent across four metrics, which is reassuring but is not a replication.
• Trained at 4K. Every long-context claim is an extrapolation past the training length. The 8K column is where all the recurrent models fall apart, this one included.
• Scale behaviour unknown. The claim is about an edit rule for a finite state. Whether finer gating still pays when the state is larger, or when the model is ten times bigger, is not addressed and is the question that decides whether anyone adopts it.
• KDA underperforms here. In this reproduction KDA scores 16.81 WikiText perplexity against Gated DeltaNet's 16.40, so channel-wise decay alone is a loss in this setup. That is either a reproduction detail or a real result about decay, and the paper does not say which.
• No downstream evaluation. Commonsense and needle retrieval, no instruction tuning, no coding, no agentic use. Nothing here tells you how the layer behaves in a product.

If you are tracking how far memory can move out of a model's forward pass, this sits next to our readings of memory leaves the model and read and write memory, where the erase decision is an explicit system component rather than a learned gate. The compression framing connects to Kimi K3: where the compute went, which is the production lineage KDA came from, and the underlying architecture is still the one discussed in Attention Is All You Need, nine years on.

7. Sources

1. Ali Hatamizadeh, Yejin Choi and Jan Kautz, Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention, NVIDIA. Every table, quotation and figure above is from this paper.
2. Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen and Yoon Kim, Parallelizing Linear Transformers with the Delta Rule over Sequence Length, the DeltaNet chunkwise algorithm this builds on.
3. Songlin Yang, Jan Kautz and Ali Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule, the direct predecessor and the main baseline.
4. Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture, where Kimi Delta Attention and its channel-wise decay are introduced.
5. Tri Dao and Albert Gu, Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, the Mamba-2 baseline.
6. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas and François Fleuret, Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, the linear attention formulation in section 1.
7. Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber, Linear Transformers Are Secretly Fast Weight Programmers, the fast-weight reading of the update.
8. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang and Boris Ginsburg, RULER: What's the Real Context Size of Your Long-Context Language Models?, the needle-in-a-haystack suite in section 3.

What we did not verify: all of it, empirically. We have not run the code, retrained any baseline, or checked a single number against a second source. The equations above are our transcription of the paper's and may differ in notation from the original. The break-even arithmetic in section 5 is ours and derives only from the dimensions the paper states. The positions in sections 4 and 5 are arguments we are making from the authors' tables, not claims they make, and the attribution of the gain to the erase gate rests on a single ablation row at a single scale. Argue with it.

This note sits in our Research track, alongside the foundation models thread. The full library is at Research.