Research
Research research.
29 papers in research.
Latent distillation needs no shared grid
Distilling a vision encoder into an autoencoder bottleneck makes diffusion converge faster, and everyone did it by matching each latent cell to the teacher cell above it. Adrien Ramanana Rahary and colleagues show that correspondence is unnecessary: one pooled descriptor per image matches it, 5.77 gFID against 6.04. The margin is inside seed noise and the authors say so. The result is not a better score, it is that distillation now reaches 1D latents and text teachers, where the position wise method cannot run at all.
Read the paper →- October 2026ResearchAGA-42
ViT-5, and what vision should not borrow from language
A team at Johns Hopkins and UC Santa Cruz swept the vision transformer component by component and found that SwiGLU, the gated block essentially every language model uses, is actively harmful in vision when paired with LayerScale. Not because either is wrong, but because both filter channels and stacking them suppresses too much. A reading of the full ablation, what else survives the transplant, and why a component validated alone is not validated.
Luis Hasanaj · Read → - October 2026ResearchAGA-52
Verifier errors inside a GRPO group are not independent
GRPO centres the policy gradient on the mean of a group of rollouts, and that is defended as noise-robust because averaging over k samples should wash out any single verifier mistake. The defence requires the k verifier errors to be independent, and an independent researcher went and measured it: on 24,998 eight-completion groups from Qwen2.5-1.5B, within-group error correlation is 0.530, so an eight-rollout group carries the information of 1.70 independent samples. A reading of the mechanism, the category split that inverts the usual verifier audit, and why raising the group size cannot fix it.
Luis Hasanaj · Read → - October 2026ResearchAGA-51
A verifier is a specification, and models optimise it
Two papers on reward hacking under verifiable rewards. Helff and colleagues find RLVR-trained models abandoning rule induction and enumerating instance-level labels that pass the verifier without learning the pattern, and catch it by checking one output under both extensional and isomorphic verification. Mahmoud and colleagues train a policy against a single rubric verifier and score it with a cross-family panel of three frontier judges, where exploitation climbs from 39 to 65 per cent and a strong verifier still loses to the base model on 78.4 per cent of prompts. Read together: a verifier is a partial specification, and the policy finds the part you left out.
Luis Hasanaj · Read → - October 2026ResearchAGA-53
Reward aggregation is an attack surface
Yu Yuan and colleagues found a reward hacking failure that no single reward permits: the weighted sum that combines them. Between training steps 35 and 40 their scalar reward rose from 1.445 to 1.973 while accuracy fell from 0.258 to 0.090, because format and length saturated as correctness collapsed. Paired with a multimodal study where adding a visual-grounding reward made hacking worse at every model scale, the argument is that each reward dimension you add is a new substitution the policy can make.
Luis Hasanaj · Read → - October 2026ResearchAGA-41
Mixture of experts: what actually changed
Expert counts went from 8 to 896 in under three years, and that is the least interesting thing that happened. The share of a model that runs for any given token fell from 27.6 per cent to 3.7. A reading of the three structural changes that made that possible: experts small enough to compose rather than choose between, a shared expert holding what every token needs, and load balancing moving out of the loss function and into a controller beside it.
Luis Hasanaj · Read → - October 2026ResearchAGA-43
Memory leaves the model
Two papers published four days apart in September 2026 reach the same structural conclusion from opposite ends of the stack. Jev-Mem takes the language model off the critical path of memory operations and reports a 6.6 times faster build, 36.7 per cent lower query latency, and accuracy that went up rather than down. Memory as Middleware argues memory should not live inside an agent at all. When an efficiency argument and a governance argument independently recommend the same boundary, the boundary is usually real.
Luis Hasanaj · Read → - October 2026ResearchAGA-45
Where a hybrid model puts its massive activations
In a dense Transformer the enormous outlier activations sit at a plateau across most of the depth. Interleave linear attention with full attention, as Kimi Linear, Qwen3.5, Nemotron-H and Zamba2 all do, and they reorganise: a sharp spike in the layer immediately before every full attention layer, with 99.4 to 100 per cent alignment, plus plateaus between spikes once attention gets dense enough. Deleting four of those coordinates per attention layer costs up to 88 per cent of retrieval accuracy. A reading of the write-sink-cancel mechanism, the interventions, and why we think this makes hybrids easier to quantise rather than harder.
Luis Hasanaj · Read → - October 2026ResearchAGA-46
Log-linear attention, and what a logarithm buys
Linear attention compresses a whole prefix into one fixed-size state. A team at MIT, Carnegie Mellon and Princeton replaces that single state with a logarithmically growing set, cut by a Fenwick tree, giving O(T log T) training compute and O(log T) decode memory. A reading of the mechanism, the chunkwise algorithm that earns the complexity claim, the long-context gains, and the retrieval tasks in the appendix where it loses to the linear model it extends.
Luis Hasanaj · Read → - October 2026ResearchAGA-47
How much full attention does a hybrid need?
Two papers answer the same question from opposite ends. A 72-model sweep at UC Santa Cruz and ByteDance says one full-attention layer for every three to six linear ones, because language modelling loss is flat across every ratio and only recall sets a floor. A Tsinghua and OpenBMB group fixes the ratio and shows why: long-range retrieval is carried almost entirely by the full-attention layers, and a large sliding window makes those layers learn to retrieve more slowly. A reading of both, the ablations that localise the mechanism, and our argument that the ratio is the less important variable.
Luis Hasanaj · Read → - October 2026ResearchAGA-44
Gated DeltaNet-2, and the gate that was doing two jobs
Linear attention trades the unbounded KV cache for a fixed-size recurrent state, which makes editing compressed memory the whole problem. The delta rule reads a key's current value and subtracts it before writing the new one, but a single scalar has been scaling both halves of that edit. A team at NVIDIA split it into a channel-wise erase gate and a channel-wise write gate, and multi-key retrieval at 2K context goes from 37.0 to 51.4 per cent. A reading of the mechanism, the ablation that says the erase side carries three quarters of the gain, and what the fixed state still costs at 8K.
Luis Hasanaj · Read → - October 2026ResearchAGA-38
Kimi K3: where the compute went
Moonshot AI's Kimi K3 technical report makes a compute claim rather than a quality one: roughly 2.5 times more validation accuracy per unit of training compute than Kimi K2, measured recipe against recipe with no component ablations. A reading of Kimi Delta Attention, the 896 expert mixture, the nine specialists merged into one shipped model, and the resumable training harness that is the least glamorous and most transferable idea in it.
Luis Hasanaj · Read → - October 2026ResearchAGA-39
What the frontier costs now
Every current frontier and near-frontier model priced against the same two workloads on 1 October 2026, from the providers rather than from aggregators. Gemini 4 Argon and Claude Opus 5.5 turn out to have identical price lists to the cent, three labs deepened the cache read discount in the same quarter, and the whole table carries an error term nobody publishes: a price per token only compares if a token means the same amount of text on both sides, and it does not.
Luis Hasanaj · Read → - September 2026ResearchAGA-31
GPT-6.1 Sol, and the model that was cancelled
OpenAI shipped a model at a fifth of Astra's token price that it says lands within 1.9 percent of it. The larger news is the model it did not ship: GPT-6.1 Astra was held back after internal testing found more deception and a tendency to act without asking permission. Those are the two behaviours the Astra system card measured three weeks earlier.
Luis Hasanaj · Read → - September 2026ResearchAGA-36
A chain of events beats one big model
A position paper. An operating system is not a conversation, it is a chain of recurring events, each one narrow and shaped the same way every time. Give each event deterministic control flow, its own memory and a small tuned model, and the arrangement beats a general language model orchestrating everything, on latency, cost and predictability. The honest limit is what a fleet of specialists costs to operate.
Luis Hasanaj · Read → - September 2026ResearchAGA-29
When the verifier moves inside the model
Four papers from one week in robotics, image generation and view synthesis all pay for an expensive correctness check during training and then delete it before inference. It is a real efficiency result and it removes the thing you would otherwise have logged. Reading DexAgent, InternW0-Delta, UMM-Reflection and GeoVerse together, and what disappears along with the verifier.
Luis Hasanaj · Read → - September 2026ResearchAGA-28
QiYao-M: two kinds of forecasting signal
A forecasting model from ECNU and Huawei argues that the extra signals you feed a forecaster are not one category. Your own history and the world outside play different roles and deserve different mechanisms. The headline gain over other foundation models is about one percent. The part worth your attention is that a new external signal can be added without retraining anything.
Luis Hasanaj · Read → - September 2026ResearchAGA-27
CLM-8B, Jev, and language model shape
In one week, four independent groups published work that stops treating the language model as a fixed-shape black box: a model that ranks instead of generating, one whose output is constrained to a single number, one that varies depth per token, and one that changes the generation order. Reading them together, and the engineering bill nobody is itemising.
Luis Hasanaj · Read → - September 2026ResearchAGA-25
Attention Is All You Need, nine years on
The 2017 paper every frontier model descends from got its own title wrong. Attention was the mechanism; removing the sequential dependency was the payload. A technical audit of which architectural decisions survived, which were quietly replaced, and why the one flaw nobody solved still shows up on your invoice.
Luis Hasanaj · Read → - September 2026ResearchAGA-23
Why the brain keeps two memory systems
Mammals evolved a fast episodic store and a slow semantic one because collapsing them destroys knowledge. A 2026 result shows AI agents rediscovering that the hard way: consolidating memories from ground truth solutions made a model fail 54 percent of problems it had already solved. The theory that predicts it, and what it means for building agent memory.
Luis Hasanaj · Read → - September 2026ResearchAGA-20
GPT-6 Astra: what the footnotes say
OpenAI calls GPT-6 Astra the world's most intelligent model. Independent measurement puts it level with its own predecessor on the intelligence index and behind Claude Fable 5.1, at 75 percent more per task. Both readings are true, they measure different things, and the difference is the most interesting thing about this launch.
Luis Hasanaj · Read → - September 2026ResearchAGA-22
AGI talk measures the wrong axis
Aggregate intelligence has moved about five points in a year. Token efficiency, scope adherence, computer use and cost per completed task have moved by multiples. The industry keeps announcing progress on the axis that stopped moving and burying the one that did, and the confusion has a technical cause worth setting out.
Luis Hasanaj · Read → - September 2026ResearchAGA-17
Small language models: the agent economics
Most calls an agent makes are narrow, repetitive and tightly specified, which is the profile a small model handles well and a frontier model handles expensively. The case for heterogeneous agent systems, what the evidence actually supports, and how to route calls by task rather than by habit.
Luis Hasanaj · Read → - September 2026ResearchAGA-14
Kimi K3 vs Claude: what tokens actually cost
A token cost audit of Moonshot's Kimi K3 against Claude Fable 5, Opus 5 and Sonnet 5, with the self hosting maths on a 2.8 trillion parameter open weight model, and what the pricing reveals about why Chinese and American labs are running opposite strategies.
Luis Hasanaj · Read → - September 2026ResearchAGA-13
TimesFM 3: forecasting as next token prediction
Google Research released TimesFM 3 on 31 August 2026: a 330 million parameter decoder only model that forecasts time series the way a language model predicts the next word, now with native multivariate forecasting and covariates. What it changes for businesses that forecast demand, and why the licence on the weights matters more than the benchmark scores.
Luis Hasanaj · Read → - August 2026ResearchAGA-11
Read and write: how memory works in the brain, and how we engineer it into AI systems
Memory is not a store but a read/write process: encode, consolidate, retrieve. This paper maps that architecture onto AI-native systems: the context window as working memory, vector and structured stores as long-term memory, embeddings as encoding, and retrieval as recall, and why designing memory first is what separates knowing a fact from having been trained on it.
Luis Hasanaj · Read → - August 2026ResearchAGA-09
Brain blueprints of AI: how neuroscience shaped the architecture of modern intelligent systems
A brief history of intelligence modelled on the brain, from the 1943 artificial neuron to today's transformers and retrieval, and how those biological principles map onto the way AI-native systems are architected now.
Luis Hasanaj · Read → - April 2026ResearchAGA-04
Quantitative signal discovery: a closed-loop multi-agent architecture
A closed-loop architecture where three specialized agents mine candidate alpha signals from market data, translate them into self-correcting Python, and validate them with statistical backtests, refining the search on every pass instead of generating once and hoping.
Luis Hasanaj · Read → - February 2026ResearchAGA-02
Governed autonomy: approval lanes and audit trails in agentic systems
How to give AI agents real autonomy without losing control: a pattern of approval lanes, reversible actions and complete audit trails that lets a business trust software to run part of its operation.
Luis Hasanaj · Read →