How much full attention does a hybrid need?

This reads two papers by other people. The first is A Systematic Analysis of Hybrid Linear Attention by Dustin Wang, Rui-Jie Zhu, Steven Abreu, Yong Shan, Taylor Kergan, Yuqi Pan, Yuhong Chou, Zheng Li, Jibin Wu, Ge Zhang, Wenhao Huang and Jason Eshraghian, at UC Santa Cruz, ByteDance Seed, the University of Groningen, CASIA and PolyU. The second is Rethinking the Role of Efficient Attention in Hybrid Architectures by Ziqing Qiao, Yinuo Xu, Chaojun Xiao, Zhou Su, Zihan Zhou, Yingfa Chen, Xiaoyue Xu, Xu Han and Zhiyuan Liu, at Tsinghua University and OpenBMB. The work and every figure below are theirs. We have not trained or reproduced either one. What is ours is the reading: putting the two side by side, and what the pair implies for anyone choosing a ratio.

Almost every long-context model shipping now is a hybrid. Some layers run full softmax attention with a growing KV cache. The rest run something cheaper: a sliding window, or a linear recurrence with a fixed-size state. The ratio between them is the single most consequential number in the architecture, because it sets the KV cache, and the KV cache sets what inference costs at long context.

Two papers attack that number from opposite ends. One sweeps the ratio and holds the cheap layer fixed. One holds the ratio fixed and interrogates the cheap layer. Read together they give a defensible answer, and they disagree in a way that is more useful than either answer alone.

1. The ratio, up front

Wang and colleagues trained 72 models to find it. Thirty-six at 340M parameters on 20B tokens, thirty-six at 1.3B on 100B. Each set crosses a family of linear attention variants against five hybridisation ratios of linear to full layers: 24:1, 12:1, 6:1, 3:1, and pure linear with no full attention at all. The variants span three generations of the idea, from HGRN and Hawk's gated vector recurrence, through RetNet's fixed decay and GLA's data-dependent gates, to DeltaNet and Gated DeltaNet's delta-rule updates.

Their recommendation is specific:

Employ a gated, hierarchical backbone (e.g., HGRN-2 or GatedDeltaNet) with one soft-max attention layer for every 3-6 linear layers.

That buys near-Transformer recall while "shrinking the KV cache by a factor of 4-7".

2. The metric that hides the answer

The reason a ratio sweep was needed at all is that the obvious metric is blind to it.

Across every configuration, language modelling accuracy barely moves. At 1.3B every architecture lands between 0.547 and 0.565 average on the standard suite of ARC-c, ARC-e, HellaSwag, LAMBADA, OBQA and PIQA, with Gated DeltaNet at 24:1 and HGRN-2 at 6:1 tied at the top on 0.565 against the Transformer's 0.548. At 340M the whole table sits lower and just as tight, the best configuration being HGRN-2 at 6:1 on 0.456 against a full-attention Transformer baseline of 0.444. A 24:1 ratio and a 3:1 ratio look the same through this lens.

Recall does not look the same. On RULER, which tests retrieval, multi-hop tracing, aggregation and question answering, the 1.3B models go from 0.256 average at pure linear to 0.397 at 3:1. The best individual results sit at Gated DeltaNet 3:1 on 0.436 and HGRN-2 6:1 on 0.434, against the full-attention baseline's 0.423. Below a 3:1 ratio the curve is steep. Above 6:1 it is flat and already paid for.

This is the part worth internalising even if you never train a model. The quantity that degrades when you make attention cheaper is not fluency. It is the ability to go and get a specific thing from far away. Loss will not tell you that you broke it.

3. Standalone scores do not predict hybrid scores

The second finding in the paper is sharper than the ratio, and gets less attention.

The relative performance of different architectures changes substantially when linear and full attention mechanisms are combined, making standalone scores unreliable predictors of hybrid performance.

Gated DeltaNet leads as a standalone linear model. HGRN-2 wins once hybridised. Which means the entire literature of linear attention leaderboards, where variants are compared to each other in isolation, is measuring the wrong configuration for the way anyone actually ships. We made the same structural argument about vision components in ViT-5, and what vision should not borrow from language: a component validated alone is not validated.

The paper attributes the hybrid advantage to three properties. Selective gating: models that expose hidden state to a learned token-wise gate "consistently perform best in recall once hybridised, surpassing the Transformer baseline by 2-5 percentage points at the optimal ratio". Hierarchical recurrence: HGRN-2's two-level pathway keeps coarse summaries on a slow update and token detail on a fast one, and the authors say this hierarchy "doubles recall". Controlled forgetting: both winners have an explicit mechanism to prevent unbounded state accumulation, Gated DeltaNet through an outer-product delta rule, HGRN-2 through gated diagonal decay.

4. The second paper asks what the cheap layer is for

Qiao, Xu and colleagues do not sweep the ratio. They fix it at 1:1, alternating full and efficient layers, and vary what the efficient layer is: sliding windows of 128, 512 and 2,048 tokens, plus Lightning Attention, Mamba-2 and Gated DeltaNet. Five model scales from 15M to 477M non-embedding parameters, token budgets from 100 to 1,000 times parameter count, pretrained directly at 16K context.

Their first result is that the choice barely matters in the limit. Validation loss curves "closely overlap" across all hybrids. Long-context perplexity shows large gaps in the low-data regime that shrink with more training. So the efficient module does not set the ceiling on long-context ability. It sets how fast that ability arrives.

Then they localise where the ability lives, with the cleanest experiment in either paper. Cap the receptive field of one layer type at inference and see which cap hurts. Restrict the full-attention layers to roughly 2,048 tokens and log long-context perplexity "increases sharply across all hybrid models". Restrict the efficient layers to the same window and almost nothing changes.

Layer-wise probing agrees. Train a classifier on hidden states to locate a needle, layer by layer, and the gains concentrate at the middle full-attention layers. The middle efficient layers "contribute little gain and even reduce accuracy". Long-range retrieval is carried by the full attention. The efficient layers are doing local mixing and nothing else.

5. Large-Window Laziness

That leads to the result the paper is named for, and the one we find most interesting, because it is not really about attention.

If you believe the efficient layer only does local work, a bigger local window should be weakly better or neutral. It is not. SWA-2048 is the outlier in their training dynamics: its retrieval-head attention entropy stays high, and the distance of its query and key projections from their final values shrinks more slowly than for SWA-128, SWA-512, Lightning, Mamba-2 or Gated DeltaNet. The retrieval heads in the full-attention layers form later.

The explanation is a gradient argument. A large window "can already cover many useful dependencies during training", so the model "can often predict the next token using information within the sliding window, without relying on full attention to retrieve from farther positions". That removes the optimisation pressure that would have taught the full attention layers to retrieve. A small window cannot cover those dependencies, so the signal reaching the global path is denser and the retrieval heads sharpen earlier.

The authors support the premise with a gradient influence profile measured against Llama-3.1-8B: signal beyond 2,048 tokens decays to a flat baseline, while the 512 to 2,048 range "still contains substantial gradient signal". A 2,048 window sits exactly where it absorbs most of what is left.

Our position, which the authors do not state: this is a general result about mixed-cost architectures, not a result about attention. Whenever a network contains a cheap path and an expensive path that solve overlapping problems, and the training objective can be satisfied by the cheap one, the expensive one is under-trained. That is the same failure shape as a collapsing router in a sparse model, which we covered in Mixture of experts: what actually changed. The remedy is the same in both cases: starve the cheap path deliberately. Large-Window Laziness is capacity competition with a new name, and naming it per-architecture is how the field keeps rediscovering it.

6. The design fix, and its size

Mechanism in hand, they make one change: apply NoPE, no positional encoding, to the full-attention layers only, in a small-window SWA hybrid. The reasoning is that if the local layers already carry position, the global layers should not be biased toward nearby tokens.

At 0.22B parameters and roughly 100B tokens:

At 0.66B:

Read against the hybrid it modifies, which is the comparison that isolates NoPE, needle retrieval goes from 49.58 to 67.81 at the smaller scale and 65.91 to 82.31 at the larger, for a change that costs no parameters and no compute. Short-context average moves 0.15 points down in one case and 0.01 up in the other, which is noise. The Full column is the more flattering baseline and we have not used it here: against it the gain is 31.86 points, but that figure pays for the window change too, and only the second half of it is NoPE.

7. What we take from reading them together

The two papers do not contradict each other, but they rank the design variables differently, and the ranking is the useful output.

On the ratio, they roughly agree, from different directions. Wang's sweep says 3:1 to 6:1 linear to full. Qiao's design section reports that moving from 1:1 to 1:3 full to efficient reaches the same validation loss. Both are saying the full-attention budget is small but strictly non-zero, and that the floor is set by retrieval rather than by perplexity. If you want one number to start from, one full-attention layer in four is inside both papers' safe range.

Our position: the ratio is the less important variable. Wang's best configuration beats the Transformer baseline on recall by 0.013 absolute, at 0.436 against 0.423. Qiao's window and encoding change together move needle retrieval by 31.86 points absolute at 0.22B against the full-attention baseline, from 35.95 to 67.81, an 89 per cent relative gain, and the cheaper configuration is the one that wins. A team with one change to spend should spend it on the local window and the positional encoding, not on the ratio. The ratio is a cost decision dressed as a quality decision.

And a caution on the ratio evidence itself. Wang's recall numbers come from RULER run at a 2,048 token context window, which the authors state plainly as a limitation. RULER was built to probe long context; at 2K it is testing recall inside a short window. We would read that recommendation as a KV-cache recommendation with a recall sanity check attached, not as a long-context finding. The long-context evidence in this pair belongs to the second paper, and the second paper did not sweep the ratio carefully. Nobody has yet run the full cross of ratio against window at a real context length, and that is the experiment we would want.

8. What neither paper establishes

Both author groups are unusually direct about this, which is worth saying.

Wang's analysis is "confined to models up to 1.3B parameters, a 2,048-token context window, and block-wise mixing ratios". Whether the trade-offs hold at 10B or above, at 128K context, or under instruction tuning and multilingual data is explicitly open. Block-wise mixing also means their ratios are layer patterns, not head-level or within-layer mixes, which is how several shipping models actually do it.

Qiao's largest model is sub-billion with at most around 100B tokens, "smaller than the scale of frontier industrial systems". They pretrain directly at 16K and extend to 32K, which inverts the standard recipe of short-context pretraining followed by extension, so the emergence timings may not transfer. Their coverage excludes RWKV-7 and Kimi-Linear. And they flag that their design experiments "are intended to validate our mechanistic conclusions rather than to serve as a full design study", so the NoPE result is a demonstration, not a tuned recommendation.

Two further gaps that neither states. Neither paper reports throughput or latency, so the cost side of the cost-quality trade is assumed rather than measured. And both measure retrieval almost entirely through synthetic needle tasks plus LongBench. Retrieval of a planted string is not the same capability as reasoning across a long document, and a mechanism localised with needles may not be the mechanism that carries the harder thing.

9. Sources

1. Dustin Wang, Rui-Jie Zhu, Steven Abreu, Yong Shan, Taylor Kergan, Yuqi Pan, Yuhong Chou, Zheng Li, Jibin Wu, Ge Zhang, Wenhao Huang and Jason Eshraghian, A Systematic Analysis of Hybrid Linear Attention. UC Santa Cruz, ByteDance Seed, University of Groningen, CASIA and PolyU. Every figure and quotation in sections 1 to 3 is from this paper.
2. Ziqing Qiao, Yinuo Xu, Chaojun Xiao, Zhou Su, Zihan Zhou, Yingfa Chen, Xiaoyue Xu, Xu Han and Zhiyuan Liu, Rethinking the Role of Efficient Attention in Hybrid Architectures. Tsinghua University and OpenBMB. Every figure and quotation in sections 4 to 6 is from this paper.
3. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang and Boris Ginsburg, RULER: What's the Real Context Size of Your Long-Context Language Models?, the recall suite both papers report against.
4. Songlin Yang, Jan Kautz and Ali Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule, the controlled-forgetting mechanism discussed in section 3.
5. Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun and Yiran Zhong, HGRN2: Gated Linear RNNs with State Expansion, the hierarchical recurrence that wins once hybridised.
6. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das and Siva Reddy, The Impact of Positional Encoding on Length Generalization in Transformers, where NoPE comes from.
7. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng and Yao Fu, Retrieval Head Mechanistically Explains Long-Context Factuality, the retrieval-head notion section 5 traces through training.
8. Our own Attention Is All You Need, nine years on for the quadratic cost both papers are working around, and Kimi K3: where the compute went for how a shipping model spends its attention budget.

We have not trained, fine-tuned or reproduced either paper, and we have not re-run any benchmark named above. Every number is as reported by its authors, read from the arXiv versions linked. We did not verify the RULER or LongBench harness configurations behind either paper's scores, which matters because both suites admit several and the numbers are not comparable across them. We have not independently checked that the two papers' ratio recommendations are measured on compatible definitions of a layer. Where we have drawn a conclusion the authors do not state, which is the Large-Window Laziness generalisation in section 5 and most of section 7, it is labelled as ours and should be argued with.