Latent distillation needs no shared grid

This paper reads Diffusable Latents from Structure-Agnostic Distillation by Adrien Ramanana Rahary, Nicolas Dufour, Patrick Pérez and David Picard, presented at the Principles of Generative Modeling workshop at NeurIPS 2026. The work and every figure below are theirs. We have not reproduced it. What is ours is the reading: which result matters, why the headline number is not it, and what the finding implies for anyone building a latent space.

Latent diffusion trains a diffusion prior inside an autoencoder's latent space, and how good the samples get depends on how diffusable that latent is. The established way to improve it is to distil a frozen pretrained vision encoder into the autoencoder bottleneck, and the established way to do that is to match each latent cell to the teacher patch sitting above it.

That requirement to sit at the same position has been carried along without much argument. It is intuitive, it works, and it quietly fixes the student's layout to the teacher's.

This paper removes it. One pooled descriptor per image, no token correspondence at all, and the result holds. The number everyone will quote is 5.77 gFID against 6.04 for the position wise baseline, and that number is the least interesting thing here. The authors say as much themselves, which is to their credit, and the real finding is somewhere else.

1. What the constraint was

The baseline is VA-VAE's vision foundation alignment, written VF throughout. It takes each cell of the autoencoder's latent grid and aligns it to the co located feature from a frozen teacher. Latent cell at row three column five is pulled toward teacher patch at row three column five.

For that to be defined at all, the latent and the teacher need a shared spatial grid. If the student's bottleneck emits a flat set of tokens with no grid, there is nothing to align to. If the teacher is a text encoder, there is no grid on the teacher's side either.

So the constraint is not only a design preference. It is a precondition, and it rules out whole classes of student architecture and teacher modality before any experiment runs.

2. Three ways to ask for less

The paper sweeps a single axis: how much correspondence the objective demands.

Every structure agnostic objective starts the same way. For each image in a batch, mean pool the student's latent tokens into one L2 normalised descriptor, and do the same on the teacher's side. Pooling discards the token layout, which is the whole point: the same objective now works whatever shape the latent is.

Pool-Align is first order. A learned linear map sends the student descriptor into the teacher's space, since the two dimensions generally differ, and the loss is the cosine distance between them. One tie per image.

CKA and Soft-KL are relational, and tie nothing at all. They compare images only through the batch by batch similarity matrix built from the normalised descriptors. Because a similarity matrix is invariant to the space it was computed in, neither needs a projector. CKA matches the two matrices as a whole with the unbiased centred kernel alignment estimator. Soft-KL is softer: a temperature softmax turns each row into a distribution over that image's nearest neighbours, and the loss is the KL between teacher and student rankings.

Three levels, then: every token tied, one descriptor tied, nothing tied.

3. The headline number is not the result

On setting A, where the latent is a 2D grid and the teacher is DINOv2, all five methods apply and the comparison is like for like.

Pool-Align wins by 0.27, and 0.13 under classifier free guidance, where the figures are 1.99 against 2.12.

Now read the authors' own appendix on significance. They cite a coefficient of variation near 1.3 per cent for gFID across training and sampling seeds, which at these operating points is roughly 0.1 to 0.2 FID, and they describe that as the smallest difference worth reading. Their margin over VF is 0.27, from one run per objective.

Their conclusion, stated plainly: Pool-Align is on par with or slightly better than VF.

That is the right call, and it is worth pausing on how unusual it is. A workshop paper with a number in its favour, and the authors spend an appendix explaining that the number is barely outside noise. The claim they actually make is a negative one, and negative claims are harder to sell: dropping correspondence does not cost anything.

Which is exactly what makes the rest of the paper work.

4. The result is reach

If correspondence were load bearing, removing it would cost something, and you would keep it. Since it costs nothing, you can go where it was never available.

Setting B, a 1D sequence latent. A Perceiver resampler bottleneck emits a flat set of tokens with no spatial grid. VF has nothing to align to and is marked n/a in their table. The pooled objectives run unchanged.

From 26.49 to 15.62. Against a noise floor of 0.1 to 0.2, that is not a margin anyone needs an appendix to defend.

Setting C, a text teacher. The latent is a 2D grid again, but the teacher is BGE-large embedding per image captions. No grid on the teacher's side, so VF drops out again.

Distilling a text encoder into an image autoencoder takes gFID from 9.52 to 6.97, which lands close to the 5.77 that the image teacher achieved. Whatever the text encoder knows about these images transfers into the latent through nothing but a pooled vector and a linear map.

That result sits interestingly against ViT-5, and what vision should not borrow from language, where transplanting SwiGLU out of language models and into a vision transformer measurably hurt. The two are not in conflict, because they operate at different levels. Borrowing a language model's components into a vision backbone can break it. Borrowing a language model's supervision through a pooled descriptor evidently does not, which suggests the transferable thing is what the teacher knows rather than how it is built.

And note CKA at 10.41. That is worse than not distilling at all. The strictest of the three constraints, reproducing the teacher's full centred between image geometry, is the one that breaks when the modality changes. The authors flag it rather than dropping it, and the ordering across the three settings is the paper's most useful internal signal: the less structure an objective demands, the further it travels.

5. Why dropping correspondence might help

The authors offer a hypothesis rather than a proof, and it is a good one.

A self supervised vision transformer's tokens do not carry only content. They carry a substantial positional signal, and high norm artefact tokens besides. Position wise alignment copies the whole per token feature into the latent, positional component included, where it occupies capacity and competes with the content the decoder has to reconstruct.

Pooling averages across tokens, and the positional part largely cancels.

Their latent PCA is consistent with this. The undistilled latent is nearly featureless. The position wise VF latent carries a smooth horizontal spatial gradient, an artefact inherited from the teacher's positional embeddings. The pooled latents show no such gradient and instead separate foreground object from background.

So the mechanism on offer is not that pooling adds something. It is that position wise alignment was importing a distractor along with the signal, and that the cost of that distractor was roughly equal to the benefit of the extra supervision. Two effects of similar size cancelling is a reasonable explanation for a 0.27 margin that sits on the edge of noise.

It is a hypothesis. The PCA is suggestive and the positional signal literature supports the premise, but the paper does not isolate the positional component and show the margin move.

6. Why this matters outside computer vision

The pattern generalises past diffusion, which is why we read it.

Distillation is the same move as adaptation, made for a different reason, which we argued in the fine tuning paper: the valuable input in both cases is a correctly labelled record of behaviour, not compute. A pooled objective says something sharper about that record. Only a summary of it had to survive, which is a much weaker requirement than copying the teacher cell by cell.

A constraint gets adopted because it is the natural way to express an idea. Matching each cell to the cell above it is the obvious reading of "make the student like the teacher". It works, so nobody tests whether it was necessary, and because it works nobody notices that it has also been fixing the student's architecture to the teacher's.

The cost of an unnecessary constraint is rarely visible as a worse number. It is visible as an absent row in a table. The 1D latent and the text teacher are not improvements over VF; they are settings where VF has no entry.

We made a structurally identical argument about the 2017 transformer in Attention Is All You Need, nine years on: almost every specific choice in that paper was replaced, and what survived was the shape, not the settings. The same question is worth asking of anything you have inherited and not tested. Which of these constraints is load bearing, and which is just how the first person to get it working happened to write it down.

7. What this paper does not establish

The limits are real and mostly stated by the authors.

It is a four page workshop paper. One run per objective, one architecture, one recipe: 50 autoencoder epochs on 256 by 256 ImageNet at batch 256, then a LightningDiT-XL prior for 64 epochs at batch 1024. No per arm tuning of the guidance scale.

Absolute gFID differs across bottleneck structures, so the settings cannot be compared with each other, only within themselves. The paper says this.

The cross modal result rests on one text encoder and one caption source, and the authors note that their captions largely restate each image's ImageNet class, so some of that gain may be class level semantics rather than anything linguistic. They do not test text to image generation, which is the setting where a text teacher would most plausibly earn its place.

And the mechanism is a hypothesis with a PCA figure behind it, not a controlled ablation.

None of that undermines the result, because the result is modest and precisely stated: spatial correspondence is not required, and removing it extends distillation to places it could not previously go. The authors did not oversell it. We would rather read ten of these than one paper claiming a new state of the art from a 0.27 margin.

8. Sources

1. Ramanana Rahary, Dufour, Pérez and Picard, Diffusable Latents from Structure-Agnostic Distillation, arXiv 2609.39657, 30 September 2026. Principles of Generative Modeling workshop, NeurIPS 2026. All figures quoted above are from this paper.
2. Code and blog post from the authors: github.com/AdrienRR/structure-agnostic-distillation and the Kyutai write up.
3. Yao and Wang, VA-VAE, the position wise baseline and the autoencoder and LightningDiT implementation every arm builds on.
4. Oquab et al, DINOv2, the image teacher; Xiao et al, BGE-large, the text teacher; Jaegle et al, the Perceiver resampler used for the 1D bottleneck.
5. Kornblith et al, the unbiased centred kernel alignment estimator used for the CKA objective.

This note sits in our Research track, alongside the generative and multimodal threads. The full library is at Research.