Type
Benchmark.
Measurement: what a published score is actually testing, how it was produced, and how far it should be trusted.
7 papers · Type
Timestral vs TimesFM 3: a real lead, smaller than the headlines
Mistral's Timestral beats Google's TimesFM 3 on GIFT-Eval. We recomputed the comparison from the leaderboard's raw files: about two percent, on 72 of 97 datasets, worse on transport and level on sales, and overtaken on one ordering within a week. The bigger differences are elsewhere: quantiles out to the 99th percentile, CPU inference, and weights a business below a revenue line can actually put into production.
Read the paper →- October 2026ResearchAGA-58
Kimi K3.1 Pro: one benchmark, three numbers
Kimi K3.1 Pro launched on 23 October 2026 with 2.8T total parameters, 104B active, a 1M context and a third off the rate card, and it has three different published scores on Terminal-Bench 4.0 from its own launch week: 65.7 per cent, 52.8 per cent and 67.2 per cent. The spread is wider than the gap between the top five models. Claude Opus 5.5 moves 6.8 points between the two harnesses and the ordering inverts. A reading of what a launch benchmark is a number of, why the harness is part of the measurement, and why cost per task did not move when every rate fell by a third.
Luis Hasanaj · Read → - October 2026ResearchAGA-52
Verifier errors inside a GRPO group are not independent
GRPO centres the policy gradient on the mean of a group of rollouts, and that is defended as noise-robust because averaging over k samples should wash out any single verifier mistake. The defence requires the k verifier errors to be independent, and an independent researcher went and measured it: on 24,998 eight-completion groups from Qwen2.5-1.5B, within-group error correlation is 0.530, so an eight-rollout group carries the information of 1.70 independent samples. A reading of the mechanism, the category split that inverts the usual verifier audit, and why raising the group size cannot fix it.
Luis Hasanaj · Read → - October 2026ResearchAGA-51
A verifier is a specification, and models optimise it
Two papers on reward hacking under verifiable rewards. Helff and colleagues find RLVR-trained models abandoning rule induction and enumerating instance-level labels that pass the verifier without learning the pattern, and catch it by checking one output under both extensional and isomorphic verification. Mahmoud and colleagues train a policy against a single rubric verifier and score it with a cross-family panel of three frontier judges, where exploitation climbs from 39 to 65 per cent and a strong verifier still loses to the base model on 78.4 per cent of prompts. Read together: a verifier is a partial specification, and the policy finds the part you left out.
Luis Hasanaj · Read → - October 2026ResearchAGA-47
How much full attention does a hybrid need?
Two papers answer the same question from opposite ends. A 72-model sweep at UC Santa Cruz and ByteDance says one full-attention layer for every three to six linear ones, because language modelling loss is flat across every ratio and only recall sets a floor. A Tsinghua and OpenBMB group fixes the ratio and shows why: long-range retrieval is carried almost entirely by the full-attention layers, and a large sliding window makes those layers learn to retrieve more slowly. A reading of both, the ablations that localise the mechanism, and our argument that the ratio is the less important variable.
Luis Hasanaj · Read → - October 2026ResearchAGA-44
Gated DeltaNet-2, and the gate that was doing two jobs
Linear attention trades the unbounded KV cache for a fixed-size recurrent state, which makes editing compressed memory the whole problem. The delta rule reads a key's current value and subtracts it before writing the new one, but a single scalar has been scaling both halves of that edit. A team at NVIDIA split it into a channel-wise erase gate and a channel-wise write gate, and multi-key retrieval at 2K context goes from 37.0 to 51.4 per cent. A reading of the mechanism, the ablation that says the erase side carries three quarters of the gain, and what the fixed state still costs at 8K.
Luis Hasanaj · Read → - September 2026EngineeringAGA-21
GPT-6 Astra or Fable 5.1 for coding?
Both list at 10 dollars and 50 dollars per million tokens, so the headline price is a tie. The cache read rate is not, and on a coding agent that is where the bill actually lives. A head to head on the published coding benchmarks, the cost that the price list hides, and an honest answer about which we would reach for today.
Luis Hasanaj · Read →