ViT-5, and what vision should not borrow from language

This paper reads ViT-5: Vision Transformers for The Mid-2020s by Feng Wang, Sucheng Ren, Tiezheng Zhang, Predrag Neskovic, Anand Bhattad and Alan Yuille at Johns Hopkins University, with Cihang Xie at UC Santa Cruz. The work and every figure below are theirs. What is ours is the reading: which result matters, why, and what it implies for anyone assembling a system out of borrowed parts.

For three years the standard way to improve a vision transformer has been to take whatever the language models are doing and do that. RMSNorm instead of LayerNorm. Rotary position embeddings. SwiGLU instead of a plain MLP. QK normalisation. It has worked often enough that it stopped being examined.

A team at Johns Hopkins and UC Santa Cruz examined it, component by component, and found that one of the most widely copied transplants is actively harmful in vision, not because the component is bad but because of what it collides with.

1. The result

SwiGLU is the gated feed-forward block that essentially every modern language model uses. Copied into a vision transformer alongside LayerScale, which is also standard, the pair measures 83.70 per cent on ImageNet against 84.16 for the same model without the gating.

Removing SwiGLU is the single largest component gain in their ablation: +0.42 per cent averaged across model sizes, larger than removing LayerScale (0.29), dropping 2D RoPE (0.24), or reverting RMSNorm to LayerNorm (0.15).

The authors' explanation is mechanical and, once stated, obvious:

the combination of LayerScale and gated MLPs should be avoided

Both components filter channels. A gated MLP suppresses channels through its gate. LayerScale suppresses them again through a learned per-channel scale on the block output. Stack two filters in series and you get what they call over-gating: too much of the representation is suppressed, and the model is worse for it.

Neither component is wrong. The pair is wrong. And in a language model the pair is usually not present, because LayerScale is a vision convention and gated MLPs arrived from language, so the interaction only appears at the join.

2. Why this is the interesting result and not a footnote

Because of how a component normally gets adopted.

Someone tries SwiGLU in a vision transformer. They compare against the same model with a plain MLP. It helps, or it does not hurt, and it ships. What almost nobody does is check the new component against everything else already in the model, because that is a combinatorial problem and the budget is finite.

So the field accumulates parts that were each validated in isolation, and the interactions between them go unmeasured until somebody does a systematic sweep. This team did the sweep. The sweep found a collision that had been sitting in production architectures.

We have made the structural version of this argument elsewhere, in AI implementation is an architecture decision: a system assembled from individually sensible parts is not a sensible system, and the failure is always at the joins. This is the same finding at the level of a neural network block, and it is nice to see it measured rather than asserted.

3. The rest of the sweep

The full ablation, averaged across sizes, removing each component from the complete model:

Three things are worth pulling out.

RMSNorm earns its place, barely. Dropping the re-centring operation in LayerNorm "does not degrade performance" and gains about 0.2 per cent on ImageNet for the base model. A cheaper operation that is also slightly better is a good trade, and it is one of the transplants that does survive contact with vision.

Rotary embeddings need help in two dimensions. RoPE alone introduces an invariance the authors flag as undesirable: patch-level flips of an image become indistinguishable. Language is a sequence and this never arises; an image is a plane and it does. ViT-5 keeps absolute position embeddings and 2D RoPE together for exactly that reason. The transplant works only once it has been adapted to the geometry of the input.

Register tokens need positions of their own. Vanilla registers appended to the patch sequence dropped accuracy to 83.90. Giving them a separate rotary embedding with a higher frequency base restored it to 84.16. A register is not a patch and should not be positioned like one.

QK normalisation is for stability, not score. It "significantly enhanced training stability and reduced sharp loss spikes" while the accuracy gain was "modest", which is a good reason to keep it and a bad reason to report it as a win.

4. The numbers, and the honest size of them

ViT-5 is 22M, 87M, 304M and 449M parameters at small, base, large and extra-large.

Against the direct transplants, which is the comparison the paper exists to make, at large and 384px: a LLaMA-style transfer reaches 85.73, a Gemma3-style transfer 85.75, DINOv3 85.50, and ViT-5 86.00.

Be clear-eyed about that gap. It is a quarter of a point over a naive port of a language recipe. The claim is not that ViT-5 is a breakthrough; the claim is that careful per-component adaptation beats wholesale transplant by a small, consistent, repeatable margin. The segmentation gap of 2.7 mIoU over DeiT-III is the more substantial result and gets less attention than the ImageNet row, as usual.

The result we would actually plan around is none of those. Trained at 224 pixels, ViT-5 holds its performance tested as high as 512, where DeiT-III degrades badly past its training resolution. Resolution robustness is worth more in production than a quarter point on a leaderboard, because production images are not 224 pixels square.

5. What this means if you are not training a backbone

Almost nobody reading this will train a vision transformer. The transferable part is the method, and it is the same discipline we keep arriving at from other directions.

A component validated alone is not validated. The thing that made this paper worth writing is not any single choice, it is that the choices were measured together. If you adopt a library, a pattern or a model because it worked in a benchmark that did not contain the rest of your system, you have measured nothing about your system.

Borrowed solutions carry assumptions about their origin. RoPE assumes a sequence. Applied to a plane it quietly grants an invariance nobody asked for. The question to ask of anything copied across domains is not "does it work here" but "what did it assume there".

Stability and score are different wins. QK normalisation barely moves accuracy and visibly flattens the loss spikes. Plenty of changes worth making show up as fewer bad nights rather than a better number, and they are systematically under-reported because the number is what gets published.

And the one that applies directly to how the rest of this library is written: this is a paper whose main finding is negative. The most useful thing in it is that a popular component should be removed. Negative results are rarer than they should be, and the reason is that nobody is rewarded for them, which is a problem the field has not solved and this paper quietly pushes against.

The clearest companion to that is latent distillation needs no shared grid, where the finding is also negative, that the spatial correspondence everyone imposed on distillation was never required, and where the authors go further by using an appendix to argue their own headline margin sits inside seed noise. Read together, the pair makes the same point from opposite ends: one component should not have been borrowed, one constraint should never have been assumed, and both were only visible to somebody who measured instead of inheriting.

6. Sources

1. Feng Wang, Sucheng Ren, Tiezheng Zhang, Predrag Neskovic, Anand Bhattad, Cihang Xie and Alan Yuille, ViT-5: Vision Transformers for The Mid-2020s. Johns Hopkins University and UC Santa Cruz. Every figure, table and quotation above is from this paper.
2. Touvron et al., DeiT III: Revenge of the ViT, the baseline ViT-5 is measured against throughout.
3. Dosovitskiy et al., An Image is Worth 16x16 Words, the original vision transformer this modernises.
4. Shazeer, GLU Variants Improve Transformer, where the gated feed-forward block the paper recommends against in vision comes from.
5. Darcet et al., Vision Transformers Need Registers, the register tokens section 3 discusses.
6. Our own Attention Is All You Need, nine years on, for the architecture all of this is still a variation of.

We have not trained or reproduced ViT-5. Everything above is a reading of a published paper, and the figures are the authors'. Where we have drawn a conclusion they do not state, which is most of sections 2 and 5, it is our argument and should be argued with.

This note sits in our Research track, alongside the multimodal and foundation models threads. The full library is at Research.