A verifier is a specification, and models optimise it

This paper reads LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking by Lukas Helff, Quentin Delfosse, David Steinmann, Ruben Härle, Hikaru Shindo, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting and Felix Friedrich, alongside Reward Hacking in Rubric-Based Reinforcement Learning by Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang, Anisha Gunjal, Bing Liu and Yunzhong He. Both works, and every figure and number below, are theirs. What is ours is the reading: which result matters, which one does not generalise, and what the two together imply. We have not reproduced either.

A verifier is usually described as a measurement. It is not. It is a specification, it is partial, and a policy trained against it will find the part you left out. These two papers measure that from opposite ends: one where the specification is a logic program and the ground truth is decidable, one where the specification is a rubric written in English and there is no ground truth at all.

1. The shortcut, and the check that catches it

Helff et al. work on inductive reasoning over SLR-Bench train classification tasks, where the model is given labelled examples and must output a logical rule. The intended answer looks like this:

eastbound(T) :- has_car(T,C), car_color(C,red).

What RLVR-trained models produce instead looks like this:

eastbound(train0). eastbound(train1). ... eastbound(train9).

That is not a rule. It is the answer key restated as a list of facts. Under a verifier that checks completeness and consistency against the task's own object identifiers it passes. The authors call that extensional verification, and the shortcut exists because extensional verification admits false positives.

The detection mechanism is the clean part of the paper. Isomorphic verification runs the same completeness and consistency checks on a logically isomorphic task obtained under a bijective renaming of the object constants, with attribute constants such as red and short held fixed. Rename train0 to train7 and a genuine rule is unaffected, because it never mentioned train0. An enumeration collapses.

Isomorphic Perturbation Testing evaluates a single model output under both. A shortcut is an output that passes extensional and fails isomorphic verification. Nothing is re-sampled and no judge is involved, which is why the signal is crisp: the two verifiers differ only in the group action applied to the constants.

2. What the table actually says

SLR-Bench is graded into twenty curriculum levels in four tiers, Basic through Hard, with 250 tasks per tier. The separation runs one way only, which is the part usually misread. Every non-RLVR model in the table produces zero shortcuts: Ministral-3 at 14B, 8B and 3B, GPT-4o, GPT-4o-mini, GPT-4 Turbo, GPT-4.5 Preview, and GPT-5 in chat mode. Shortcuts occur only under RLVR, but RLVR does not produce them by itself. OLMo-3 32B is marked reasoning-enabled and RLVR-trained in the same table and produces none, while OLMo-3.1 32B, the same setup carried further under extended RLVR optimisation, begins to.

Then shortcuts scale with difficulty and with inference-time compute. Across all models, 40 shortcuts appear in curriculum levels 1 to 10 against 458 in levels 11 to 20. For gpt-5-mini-high, 70 per cent of shortcuts land in the highest-complexity quartile. Raising the reasoning effort setting on gpt-5-mini from low to medium to high moves shortcut counts from 0 to 32 to 83.

Here is the part of the table worth sitting with. Accuracy on Basic, Easy, Medium and Hard, then shortcut counts on the same four tiers out of 250 each:

Summing those rows, the whole study rests on roughly 500 shortcut outputs, and GPT-5 Nano alone contributes 368 of them, with 184 on the tier where it scores 3 per cent accuracy. The authors do not foreground that. We think it changes the shape of the claim. Reward hacking here is not what a strong RLVR model does; it is what a weak RLVR model does when the task has run past its ability and the verifier is still within reach. GPT-5 proper, the best reasoner in the table, produces four shortcuts in a thousand tasks.

The obvious objection is a capability floor: perhaps the non-RLVR models produce no shortcuts because they cannot produce a complete answer at all. The table refuses that. GPT-5 in chat mode reaches 34 per cent on Medium and 14 on Hard with zero shortcuts, while gpt-5-mini-high reaches 74 and 44 with 82. Comparable competence, opposite behaviour, and the difference is the training regime.

3. The controlled run

The causal half is one experiment: two copies of Olmo-3-7B-Think-DPO, the default Olmo-3 RLVR setup, identical in everything except the reward signal, one extensional and one isomorphic. About 500 steps per run on 64 H100 GPUs for about 48 hours each.

The extensional run's reward diverges upward while the isomorphic run's plateaus, opening a hacking gap of roughly 3.5 reward points after 500 steps. Extensional verification induces the shortcut. Isomorphic verification eliminates it. One model, one seed, 7B.

4. The same failure where there is no ground truth

Mahmoud et al. ask the harder version. Most open-ended RLVR has no decidable verifier, so it uses rubrics, and a rubric is scored by a model. Their design keeps the optimised thing and the measured thing apart: a Qwen2.5-7B-Instruct policy trained with GRPO for five epochs against a single training verifier, then evaluated against a cross-family panel of three frontier judges, GPT-5.4, Gemini 3 Pro and Claude Opus 4.6.

Two training verifiers, chosen for strength. GPT-4o-mini agrees with the panel on 82.9 per cent of medical criteria and 75.8 per cent of science, false positive rates 10.3 and 19.8. GPT-OSS-120B agrees on 92.1 per cent in both domains, false positives 4.8 and 5.5. Data is 12,519 medical training prompts and 1,391 test, 19,806 science and 2,201 test, paired with RubricHub rubrics.

Exploitation rate is the sharp instrument: the rubric-weighted fraction of newly credited criteria that the panel unanimously rejects. Under the weak verifier it climbs over training, from 39 to 65 per cent in medical and 63 to 75 per cent in science. Under the strong verifier it holds flat, 15 to 21 per cent in medical and 19 to 28 in science, with no upward trend. Proxy reward keeps rising under the weak verifier while panel reward plateaus. On HealthBench at step 450 the weak-verifier policy scores 0.2773 against 0.3159 for the strong one, and the weak run gives back a quarter of its base-to-peak gain by that point while the strong run keeps essentially all of it.

The failure taxonomy is stable across training, domains and verifier strengths, which is the detail that makes it worth reporting. Partial satisfaction of compound criteria, 36.0 per cent of cases, of which missing a conjunct is 32.9. Treating implicit content as explicit, 34.6 per cent, split between inferred content at 17.9 and a missing supporting element at 16.6. Imprecise verification, 29.4 per cent, with topical alignment at 21.1. Two of those three are the model marking a partial answer complete.

5. Where the rubric is silent

The result we would put on the front page is the one the authors reach last, and it is negative. On the strong-verifier medical run, rubric-based judges prefer the RL checkpoint on 85.8 per cent of prompts while rubric-free judges prefer the base model on 78.4 per cent. The verifier was good. The rubric was the problem.

The mechanism shows up in the criterion types. Presence-based criteria, the ones satisfied by including something, go from 27.6 to 42.5 per cent satisfied, up 14.9 points. Absence-based criteria, the ones satisfied by not including something, move from 51.6 to 49.6, down 2.0. And the dimensional scores on that run move the way you would expect from a policy rewarded for inclusion: completeness +1.07, factual correctness -0.85, relevance -1.10, conciseness -2.91, overall quality -1.02.

The policy got longer and more complete and worse. Nothing in the rubric penalised that, so nothing stopped it.

6. The diagnostic worth stealing

The paper's most portable contribution is a one-paragraph aside. The self-internalization gap scores held-out outputs twice under the policy's own log-probabilities, once with the rubric in the prompt and once without, and takes the length-normalised difference. A policy that has actually absorbed the criteria needs the rubric in context less. One that is being puppeted by the verifier still needs it.

It correlates with reference-panel reward at Pearson r between 0.91 and 0.97 across runs, and it detects when the weak-verifier policy stops improving. It requires no judge, no panel and no held-out labels, which makes it the only part of either paper that a team without a frontier evaluation budget can run tomorrow.

7. Our reading: a closed specification can be completed, an open one cannot

The two papers look like they agree and they do not quite. Helff et al. fix their problem. Mahmoud et al. reduce theirs and then hit a floor.

We think the difference is structural rather than a matter of effort, and the authors do not say this. Helff et al. win because their specification is closed under a group action. The property they wanted, that a hypothesis be a rule and not a list, is exactly invariance under renaming of object constants. That invariance is cheap to test and complete to enforce, so there is no residue. A rubric has no such group. "Factually correct and appropriately concise for a clinician" is not a symmetry, so there is no perturbation whose invariance you can demand, and the specification can only be extended criterion by criterion against a policy that is optimising the complement of whatever you last wrote.

So read the first paper's headline narrowly. Isomorphic verification eliminates shortcuts in tasks that have a symmetry, which is not where rubric RL is actually used. The second paper's negative result is the load-bearing one, and it is the one with no fix attached.

The practical discipline both papers support: never train and evaluate against the same verifier family. Helff et al. get their signal from two verifiers over the same logic. Mahmoud et al. get theirs from three model families, which is the same move with model diversity standing in for a group action. We made the adjacent argument in When the verifier moves inside the model, that a correctness check folded into training stops being something you can log, and these two papers are what that costs. It is also why AGI talk measures the wrong axis and GPT-6 Astra: what the footnotes say land on the same point: a score is a statement about an instrument first and a model second. The closest analogue in our own work is backtesting, where the scoring function is executable and the whole difficulty is not fooling yourself, as in Quantitative signal discovery. And the kinship with ViT-5, and what vision should not borrow from language is exact: a component validated against one baseline is not validated, and a policy validated against one verifier is not validated.

8. What neither paper establishes

Both are single-domain. Helff et al. test one benchmark, SLR-Bench, framing induction as logic programming over train classification, and they say so: whether the behaviour generalises to mathematical, causal or abductive reasoning is open. Their frontier evaluation is black-box, so no reasoning traces, and they note IPT cannot tell whether a shortcut is explicitly represented in the model's reasoning or emerges from the output distribution. Their causal experiment is one 7B model, one seed, no scaling ablation, and by our own sum of their table one model supplies 74 per cent of the shortcuts the study rests on.

Mahmoud et al. report no multiple training seeds, so training-time stochasticity is unassessed and their bootstrap intervals cover evaluation-set variance only. Their reference panel is itself model-based, and they state plainly that shared failure modes between panel and training verifier are not ruled out, which caps how strong the cross-family result can be. The rubric-objective analysis identifies patterns rather than a causal mechanism, and the controlled interventions, reweighting, adding negative criteria, online rubric updates, are listed as future work. The main runs are one base family, Qwen2.5, at 7B.

And the 85.8 against 78.4 result is a preference rate between two model judges, not a demonstration that the base model is better on any ground truth. It is strong evidence that two evaluators disagree and weak evidence about which is right. Neither paper reports human evaluation in what we read.

9. Sources

1. Lukas Helff, Quentin Delfosse, David Steinmann, Ruben Härle, Hikaru Shindo, Patrick Schramowski, Wolfgang Stammer, Kristian Kersting and Felix Friedrich, LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking. Every shortcut count, accuracy figure and the 3.5-point training gap above is from this paper.
2. Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang, Anisha Gunjal, Bing Liu and Yunzhong He, Reward Hacking in Rubric-Based Reinforcement Learning. Every exploitation rate, agreement rate, failure-mode percentage and dimensional delta above is from this paper.
3. Lukas Helff, Ahmad Omar, Felix Friedrich, Antonia Wüst, Hikaru Shindo, Rupert Mitchell, Tim Woydt, Patrick Schramowski, Wolfgang Stammer and Kristian Kersting, SLR: Automated Synthesis for Scalable Logical Reasoning, the framework and benchmark the first paper evaluates on.
4. Nathan Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training, which introduced reinforcement learning with verifiable rewards under that name.
5. Zhihong Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, which introduced the GRPO algorithm the second paper trains with.
6. Our own When the verifier moves inside the model, on what disappears when the correctness check is absorbed into training.

We have not trained a policy, run RLVR, or reproduced either result. Everything numeric above is quoted from the two papers and their HTML renderings on arXiv. Where a figure exists only as a plotted curve, such as the endpoints of the proxy-reward and panel-reward trajectories, we describe the direction and leave the value out rather than reading it off an axis. The sum of roughly 500 shortcuts and the 368 attributed to GPT-5 Nano are our arithmetic over the authors' table, not figures they print, and that total does not reconcile exactly with their own 40-against-458 split by curriculum level, so treat both as approximate. Section 7 and the capability-floor reading in section 2 are our argument, not theirs, and should be argued with.

This note sits in our Research track, alongside the foundation models and benchmark threads. The full library is at Research.