Where a hybrid model puts its massive activations

This paper reads Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus by Zunhai Su, Bohan Sun, Xialie Zhuang, Shuibai Zhang, He Xiao, Hui Shen, Jing Xiong, Hengyuan Zhang, Zhongzhu Zhou, Tiantian Zhang, Ngai Wong and Chuan-Wei Kuo. The work and every figure and number below are theirs. What is ours is the reading: which result matters, what it implies for low-bit inference, and where we disagree with the framing. We have not reproduced any of it.

A massive activation is a single entry in a model's hidden state that is larger than its neighbours by several orders of magnitude. Not twice as large: the paper's own phrase is several orders, and the signed traces bear it out. They are extremely sparse, they land on a handful of token positions, and in a dense Transformer they are largely invariant to the input. Sun and colleagues identified them in 2024 and showed they behave as implicit bias terms coupled to attention sinks.

Everything known about them was known about dense Transformers. This paper asks what happens when you interleave full attention with linear attention, which is what Kimi Linear, Qwen3.5, Nemotron-H and Zamba2 now all do. The activations reorganise into a shape the architecture dictates.

1. The two shapes

Fix a token and trace its largest absolute activation layer by layer. In a dense Transformer the trace is a plateau: the outlier appears early and stays high through most of the depth. In a hybrid it is not.

Pre-attention spikes. The magnitude rises sharply in the layer immediately before each full attention layer, then falls. The alignment is not approximate. Across 500 inputs spanning general prose, scientific writing, GSM8K, Python and multilingual text, the fraction of input-and-attention-layer pairs whose block maximum sits exactly at the layer before full attention is 99.4 to 100 per cent for all ten backbone-and-scale pairs tested: RetNet, HGRN, GLA, DeltaNet and Gated DeltaNet, at 340M and 1.3B.

Inter-spike plateaus. As full attention gets denser, the troughs between spikes fill in. Their inter-spike retention score normalises activation at the intervening linear layers by the weaker of the two neighbouring spikes, so zero is decay and one is full persistence. For 1.3B Gated DeltaNet it reads 18.4 per cent at one attention layer in twelve, 26.6 at one in six, and 77.8 at one in three. All twenty adjacent-ratio differences have paired 95 per cent bootstrap intervals above zero.

The onset is backbone-dependent, which is the detail that makes this a mechanism rather than a correlation. Plateaus appear at one-in-six in DeltaNet but only at one-in-three in Gated DeltaNet. Attention density sets the direction; the recurrent update decides where the transition lands.

The shape recurs in 12 public checkpoints from 1.2B to 397B parameters, across linear attention and state-space hybrids, base and instruction-tuned. Layerwise position is stable across post-training even where magnitude is not.

2. The mechanism, traced at one coordinate

The morphology is a description. The mechanism is the contribution.

The authors fix a single token-feature coordinate in a 1.3B Gated DeltaNet hybrid at one attention layer in twelve, and decompose the signed residual update per module. Three stages fall out. Layer 11 writes the outlier. In layer 12, full attention, that token attracts a disproportionate share of attention from later queries and acts as a sink. An opposite-signed update in the same layer substantially cancels the outlier it just used.

Write, sink, cancel. The outlier is not a persistent feature of the representation. It is scaffolding, erected one layer before it is needed and torn down immediately after.

The plateau is the same lifecycle with the cancellation delayed. Magnitudes rise before full attention, stay elevated through the intervening linear layers, and the opposite-signed update arrives when the plateau dissipates. One account, two settings of one timing parameter, and the dense Transformer is the limit where cancellation is so delayed it never visibly happens within the span.

Most outlier analysis reports where the large values are. This reports what writes them, what consumes them and what removes them, at a fixed coordinate rather than over a magnitude envelope that may be tracking different features at different depths.

3. Cutting them out

Interventions on six frozen Gated DeltaNet checkpoints, weights untouched, eleven benchmarks. At each full attention input, zero the four largest-magnitude coordinates of the first prompt token. Four numbers per attention layer.

Ten-task mean accuracy falls from 51.63 to 29.43 per cent, a 43.00 per cent relative drop, with a per-model range of 21.91 to 63.60 per cent. WikiText-103 perplexity goes from 11.6290 to 20.1000.

The distribution across tasks is the interesting part:

Commonsense reasoning barely notices. Needle-in-a-haystack retrieval collapses to near chance. Four scalars per attention layer are carrying most of the model's ability to find a specific thing in its context, which is precisely the capability hybrids are accused of trading away for efficiency.

The reverse intervention is the headline. Rather than deleting the spikes, hold them: run a reference pass, then between attention layers enforce a signed magnitude floor on the selected coordinates so a spike never decays before the next one. No retraining, no weight changes. Mean real-world retrieval across FDA, SWDE, SQuAD and NQ rises from 35.28 to 37.08 per cent, a 5.10 per cent relative gain, with 1.06 to 12.60 per cent across all eighteen model-and-budget settings. SQuAD gains 11.66 per cent relative, SWDE 6.06. Task level regressions do occur, on FDA, NQ and NIAH-3 in some settings.

The authors read this as evidence that extending persistence moves a hybrid toward the dense Transformer's morphology. We read it differently, below.

4. Why this lands on quantisation

A low-bit quantiser assigns one shared scale across a group of values. The scale is set by the largest magnitude present. One coordinate several orders of magnitude above its neighbours consumes the dynamic range, and every ordinary value in that group is left with a fraction of the available resolution.

The paper states this clearly and cites it to prior work: massive activations broaden the tensor dynamic range, a small number of them can dominate shared scales, and the result is amplified quantisation error in activation and KV cache quantisation. It is the standing motivation for outlier-aware rotations and sink-preserving KV cache methods.

It does not run a single quantisation experiment. Not one bit width, not one perplexity-under-INT8 table. The connection is argued from the literature, not measured here. That is the honest boundary of the paper, and anyone citing it as quantisation evidence is citing the wrong thing.

Our position, which the authors do not take: the morphology makes hybrids easier to quantise than dense Transformers, not harder. The existing machinery treats outliers as unpredictable and pays a global cost to handle them, because in a dense model the plateau runs through most of the depth and any layer may hold one. In a hybrid at one attention layer in twelve, the spike is at a known index, with 99.4 to 100 per cent alignment across five architectures, five domains and two scales. That is not a statistical tendency. That is an address.

The policy this implies is mixed precision keyed to attention placement: full precision at the handful of layers feeding full attention, aggressive quantisation everywhere else. Eleven of twelve layers would be cheap. We have not measured it and neither has anyone in this paper's reference list, as far as we can establish, so treat it as a falsifiable prediction rather than a recommendation. The obvious way it fails is that the plateau regime at one-in-three leaves no clean layers, which would mean the quantisation benefit runs opposite to the retrieval benefit.

5. The second disagreement

The spike-to-plateau result is presented as an improvement. We think it is better read as a diagnosis.

The method needs a reference forward pass to set the floor. That is roughly double the inference cost for a 5.10 per cent relative retrieval gain, in models at or below 1.3B. As a deployment technique it does not survive contact with a cost model, and the authors say as much when they call for reference-free variants.

What it establishes is less convenient. A hybrid's retrieval improves when its activations are forced to behave more like a dense Transformer's, which is a statement about what the linear layers fail to preserve. The efficiency argument for hybrids assumes the recurrent state is a sufficient substitute for attention between the attention layers. A 5 per cent retrieval gain from propping up four coordinates across that gap says it is not quite, and that the deficit is concentrated enough to be addressable. The follow-up worth running is not a better inference-time patch. It is whether a loss term penalising premature cancellation closes the same gap at training time, for free.

This is the same shape of argument we made about component interactions in ViT-5: a result that looks like a tuning win is often an architecture telling you where it is thin.

6. What the paper does not establish

• No quantisation measurement. Section 4's practical consequence is motivated, not tested, here.
• No overhead accounting. The connection intervention's cost is never quantified against its gain.
• Interventions stop at 1.3B. All six intervened checkpoints are Gated DeltaNet at 340M and 1.3B. The 397B observations are morphology only. Whether deleting four coordinates breaks a 397B hybrid the same way is untested.
• One token. Both interventions target the first prompt token, following the first-token analyses. Generalisation to other sink positions is not shown.
• The gating ablation has no numbers. Full attention output gating "markedly attenuates" spikes and removing Gated DeltaNet output gates gives "moderate" amplification. Both claims rest on figures. The asymmetry is the more surprising finding, since removing the recurrent gates modifies far more layers for less effect, and it is the one reported least precisely.
• The retrieval confidence intervals are labelled exploratory by the authors themselves.
• There is no limitations section. The paper has an ethics statement and a reproducibility statement. The list above is ours.

The model families here are the ones currently shipping. If you are reading hybrid architecture reports, the context is in our readings of Kimi K3, of what changed in mixture of experts and of attention itself, nine years on. The wider move to models whose shape is a variable rather than a constant is in CLM-8B, Jev and language model shape.

7. Sources

1. Zunhai Su, Bohan Sun, Xialie Zhuang, Shuibai Zhang, He Xiao, Hui Shen, Jing Xiong, Hengyuan Zhang, Zhongzhu Zhou, Tiantian Zhang, Ngai Wong and Chuan-Wei Kuo, Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus. Every figure, table and quoted number above is from this paper.
2. Mingjie Sun, Xinlei Chen, J. Zico Kolter and Zhuang Liu, Massive Activations in Large Language Models, where the phenomenon and its coupling to attention sinks were established in dense models.
3. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han and Mike Lewis, Efficient Streaming Language Models with Attention Sinks, the attention sink result the write-sink-cancel account connects to.
4. Yongqi An, Xu Zhao, Tao Yu, Ming Tang and Jinqiao Wang, Systematic Outliers in Large Language Models, the signed-update analysis this paper builds its lifecycle account on.
5. Dustin Wang, Rui-Jie Zhu, Steven Abreu and colleagues, A Systematic Analysis of Hybrid Linear Attention, the M-A-P suite supplying the 52 controlled checkpoints.
6. Kimi Team, Kimi Linear: An Expressive, Efficient Attention Architecture, one of the four public families the morphology was confirmed in.

We have not trained, quantised or reproduced any model discussed here. Every number is the authors', read from the paper. We have not verified the gating figures, because they are figures and carry no numbers to verify. The mixed-precision argument in section 4 and the reading of the connection intervention in section 5 are ours, are untested, and are the parts of this note worth arguing with.

This note sits in our Research track, alongside the foundation models thread. The full library is at Research.