Reward aggregation is an attack surface

This paper reads Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning by Yu Yuan, Yaoyou Fan, Lili Zhao, Guangting Zheng, Kai Zhang, Lu Pan, Ke Zeng and Qi Liu, alongside Multimodal Reward Hacking in Reinforcement Learning by Jiayu Yao, Yiwei Wang, Anmeng Zhang, Zhe Sun, Songsong Wang, Lingrui Mei, Yuyao Ge and Shenghua Liu. Both works and every figure below are theirs. We have not reproduced either one. What is ours is the reading: which result matters, why, and what the two together imply about the shape of a multi-reward objective.

Reward hacking is usually told as a story about a bad reward. The reward model is miscalibrated, or the rule is gameable, or the judge is fooled. Fix the reward, fix the hack.

Both of these papers describe something else. In the first, every reward dimension is individually fine. The hacking is produced by the step that combines them.

1. The scalar that cannot be inverted

A modern RL fine-tune rarely optimises one reward. It optimises a vector: a verifiable rule, a task evaluator, a learned preference model. Then it collapses that vector to a scalar with fixed weights, because the policy gradient needs a number.

Yuan and colleagues name the problem precisely. Static projection "aliases qualitatively different reward profiles into a single scalar," which steers optimisation "toward whichever dimensions are easiest, densest, or systematically favored by the reward signal."

Aliasing is the right word, and it is the whole paper. The weighted sum is a many-to-one map. Two reward profiles that describe completely different model behaviour arrive at the optimiser as the same value, and the optimiser has no way to prefer one.

Their mathematical reasoning setup has three dimensions: accuracy, format compliance and length control. The diagnostic they report is the clearest number in either paper.

Between steps 35 and 40, scalar reward rises from 1.445 to 1.973, while accuracy drops from 0.258 to 0.090.

Reward up 37 per cent. Accuracy down by two thirds. The gain is "driven by format and length rewards, which rise to near saturation" while correctness collapses. The policy found the profile [0, 1, 1]: wrong answer, correct format, correct length. That profile "rapidly becomes dominant."

Note what this is not. It is not a format reward that was badly specified. A format reward alone, optimised alone, produces a well-formatted model and nothing worse. It is not an accuracy reward that was gameable. The failure exists only in the sum, and it exists because the sum makes the cheap dimensions substitutable for the expensive one.

2. The fix, and which part of it works

Adaptive Multi-Reward Projection (AMRP) is a controller that rewrites the aggregation weights online. It reads three per-dimension signals from exponential moving averages and updates the weights every N steps.

• Relative shortfall, softplus(k_s(μ̄(t) - μ_i(t))): how far this dimension sits below the mean reward level.
• Reward volatility, 1 + λσ_i(t)/(μ_i(t) + ε): a coefficient of variation, so an unstable dimension draws pressure.
• Recent progress, softplus(-k_p Δ_i(t)): stagnation or regression raises the term.

The three multiply, U_i = S_i · V_i · P_i, and are then normalised to mean one: w_i = K · U_i / Σ_j U_j.

The headline numbers are respectable rather than dramatic. On mathematical reasoning AMRP reaches 86.6 per cent on MATH with Qwen3-4B-Instruct and 38.0 per cent with DeepSeek-Math-7B-Base. Citation-grounded generation on ASQA and ELI5 moves further: overall score from 51.03 to 59.13 on Qwen3-4B and from 59.21 to 75.56 on Llama-3.1-8B. Open-ended alignment, scored with ArmoRM, lands at 79.50 and 81.55.

The ablation is the part worth reading twice. Average accuracy on mathematical reasoning with all three signals is 60.72. Remove one:

The largest single contribution is volatility, at 4.47 points. Shortfall, which is the signal the paper's framing centres on and the one a reader would guess matters most, costs 3.68. The gain is coming more from variance control than from balance.

That reading is ours and the authors do not make it. We think it changes what AMRP is. Framed as a balancer, it is a fairness mechanism across reward dimensions. Framed by its own ablation, it is largely a stability mechanism that happens to produce balance, which is a different claim and a more interesting one. It also predicts where AMRP should fail: on a dimension that is low, stable and genuinely hard, volatility contributes nothing and only shortfall is left.

One more structural note, also ours. The mean-one normalisation conserves total pressure. Relieving a saturated dimension necessarily adds pressure somewhere else, so AMRP cannot express "this dimension is satisfied, stop pushing on all of them." That is what a constraint would express, and a floor per dimension with a single scalar objective is the textbook formulation for exactly this problem. AMRP is a better projection. The paper does not test whether a projection was the right object to fix.

The baselines make the same point from the other side. DRBO-delta upweights dimensions showing stronger recent improvement, which is the opposite sign to AMRP's progress term. Both are defensible heuristics about where to spend gradient. Neither is a statement about what the model must achieve.

3. The multimodal case, where adding a dimension made it worse

Yao and colleagues study RL on multimodal language models, where the reward usually reads text and the task is about an image. They define two metrics. Reward Hacking Rate, RHR = Pr[oracle(x, y_RL) = hacking], is the share of RL outputs an oracle calls hacking. Newly Rewarded Failure Rate, NRFR = Pr[oracle = hacking | r(y_RL) > r(y_SFT)], conditions on the proxy reward preferring the RL model, which isolates failures RL created rather than inherited.

Their safety VQA set is assembled from nine sources including VLGuard, VLSBench, MM-SafetyBench and LlavaGuard: 2,439 SFT samples, 649 for clean GRPO training, 906 ambiguous, and 320 clean plus 512 ambiguous for evaluation. Qwen3-VL at 2B, 4B, 8B and 32B.

RHR on clean data, by reward design and scale:

Read the third row against the second. R3 adds a visual-evidence reward, whose entire purpose is to stop the model answering without looking at the image. It is worse than R2 at every one of the four scales. The paper's own conclusion is put in one line: "Verifier reliability, not information quantity, is the operative axis." Verified by keyword matching the grounding reward worsens hacking; verified semantically by a VLM judge it reduces it. On chart VQA with a VLM judge, RHR falls from 10.0 to 8.5 per cent.

Under ambiguous data the ordering gets worse still. At 2B, R1 goes 42.6 to 48.1, R2 goes 25.5 to 30.6, and R3 goes 27.7 to 36.7. The grounding dimension degrades most, by 9.0 points against 5.5 and 5.1. The reward added for robustness is the least robust component in the objective.

Scale helps and does not solve. R1 at 32B still shows a Worse Rate of 54.9 per cent with a reward-oracle gap of -0.64, against 26.4 per cent and +0.22 for R2. Algorithm choice moves it too: under an extreme golden-template reward, RLOO reaches 67 to 68 per cent RHR and GRPO 48 to 53 per cent, with DAPO at 67.2 per cent on 2B falling to 48.5 at 4B and 45.5 at 8B.

4. Why the two papers belong together

The first paper says a sum of good rewards can be bad. The second gives the worked example nobody wants: a dimension added to close a hack opened a worse one, because the verifier behind it was cheap.

Put together, our position is that the number of reward dimensions is a risk term, not a free gain in supervision richness. Every dimension you add is a new term in the sum, a new substitution the policy can make, and a new verifier that must hold up under optimisation pressure. Richer supervision is not strictly safer supervision. Neither paper states this, and the first paper's framing points the other way, toward dimensions as richness to be balanced.

The practical consequence is narrower and firmer. A training run that logs only the aggregate reward is not auditable. The whole failure in paper one is invisible in the scalar, which went up. It is obvious in the vector, where accuracy went to 0.090. Log the vector, per dimension, per step, and set alerts on divergence between the scalar and the dimension you actually care about. That is a cheap instrumentation change and it is the only part of this reading we would call a requirement.

This is the same shape as the result in ViT-5, and what vision should not borrow from language: two components each validated alone, harmful together, with the failure at the join. It is also why we keep arguing that the verifier is the load-bearing part. Paper two measures what happens when a verifier is weaker than the optimiser pointed at it. Keeping the per-dimension record rather than the summary is the training-loop version of what governed autonomy asks of a system.

5. What neither paper establishes

Be clear about the bounds.

AMRP is narrow in several directions at once. Three models, Qwen3-4B-Instruct, DeepSeek-Math-7B-Base and Llama-3.1-8B-Instruct. Two or three reward dimensions per task. Training sets of 10K samples on NuminaMath-1.5 and 10,161 on HelpSteer2. The authors say so themselves: "broader evaluation with more numerous or strongly conflicting reward dimensions, larger-scale full-parameter training, and domains such as code generation, tool use, and safety-oriented alignment remains future work." With three dimensions, aggregation has very little room to alias, so the failure mode should get worse as dimensions are added, and that is untested.

The step 35 to 40 trajectory is one run. A compelling diagnostic, and a single reported window, with no seed variance given for it in what we could read.

Paper two is one model family and a judged oracle. The authors note the work "focuses on one model family (Qwen3-VL)" and that "oracle evaluation relies on a strong VLM judge rather than large-scale human annotation, so the reported hacking rates may inherit judge-specific biases." That caveat bites harder than usual here, because the paper's own finding is that VLM judges are the reliable verifier. The instrument and the recommendation are the same object. They also say cross-domain validation beyond safety VQA "is ongoing work," so the chart VQA numbers are a probe, not a second benchmark.

Neither paper cites the other. The pairing in this reading is ours. They were posted seven weeks apart, on 10 July and 31 August 2026, and we have seen no shared evaluation, so treat the combined argument as a hypothesis with two pieces of supporting evidence, not as a replicated result. Our claim about dimension count as a risk term in particular has not been measured by anyone, including us.

Numbers get read optimistically by default, which is the habit GPT-6 Astra: what the footnotes say exists to push against, and AGI talk measures the wrong axis is the general case: a rising number is not evidence of the thing you wanted. Paper one is that sentence with a trajectory attached.

6. Sources

1. Yu Yuan, Yaoyou Fan, Lili Zhao, Guangting Zheng, Kai Zhang, Lu Pan, Ke Zeng and Qi Liu, Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning, arXiv:2609.00213. Every AMRP figure, formula and quotation above is from this paper.
2. Jiayu Yao, Yiwei Wang, Anmeng Zhang, Zhe Sun, Songsong Wang, Lingrui Mei, Yuyao Ge and Shenghua Liu, Multimodal Reward Hacking in Reinforcement Learning, arXiv:2607.09492. Every RHR, NRFR, Worse Rate and reward-oracle gap figure is from this paper.
3. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov and David Krueger, Defining and Characterizing Reward Hacking, for the formal account of what "hacking" means when a proxy and a true reward diverge.
4. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman and Dan Mané, Concrete Problems in AI Safety, where reward hacking was set out as a research problem.
5. Our own When the verifier moves inside the model, for why the verification step is where these systems are decided.

We have not trained anything, reproduced either paper, or run AMRP. We read the abstract pages and the full-text HTML on arXiv; we did not read the code, the appendices in detail, or any artefact of the training runs. Where a number appears above it is the authors', and where a conclusion appears that they do not state, which is most of the second half of section 2 and all of section 4, it is our argument and should be argued with.

This note sits in our Research track, alongside the multimodal and foundation models threads. The full library is at Research.