ScaleSweep, and what AbsMax leaves on the table in NVFP4

This paper reads ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization by Li Lin and Xiaojun Wan at the Wangxuan Institute of Computer Technology, Peking University. The work and every figure and table quoted below are theirs. What is ours is the reading: which result matters, where the gain actually comes from, and which number we would plan around. We have not reproduced any of it.

Four-bit inference has a boring bottleneck. It is not the rounding of the elements. It is the scale you divide by before you round them. NVFP4 gives every 16 elements their own scale, stores that scale in FP8, and almost everybody picks it the same way: divide the block maximum by 6 and move on. This paper measures how much that costs and shows the better choice is cheap to find.

1. The format, precisely

NVFP4 is three levels of number, not one.

The elements are FP4 E2M1, two exponent bits and one mantissa bit. The representable set is small enough to write out in full, and the paper does:

{0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}

Sixteen consecutive elements share one scale held in FP8 E4M3. The whole tensor then shares a single FP32 global scale. A reconstructed value is the product of all three: the FP4 code, the block scale, the global scale.

Two things follow from that structure, and they drive everything else.

First, the grid is non-uniform. The gaps between 0.5 and 1 and between 4 and 6 are not the same size. A scale that places the block maximum exactly on 6 wastes the resolution near the top of the range, because 4 to 6 is the widest gap in the codebook. This is exactly the observation the Four Over Six work builds on, and the paper treats 4/6 as its stronger baseline for that reason.

Second, once the global scale is fixed, every block becomes an independent optimisation problem. There is no coupling left. That is what makes an exhaustive search tractable at all, and the paper is explicit that the global scale barely matters: Figure 2 sweeps it from 1x to 4x the default and normalised MSE moves within a band of about 0.04 parts per thousand. They set it to the tensor maximum over 256 times 6, following Cook and colleagues, and stop thinking about it.

The AbsMax baseline the field defaults to is the obvious thing. Take the block maximum, divide by 6, round to FP8. One candidate, no search.

2. Why the block scale is the free parameter

FP8 E4M3 has 126 positive finite values. That is the entire search space for a block scale. You could brute force it.

The paper's contribution is to prove you do not have to. It derives bounds that shrink 126 candidates to eleven.

The upper bound is the clean result. Lemma 4.1: if the scale exceeds the block maximum divided by 3.5, then halving it cannot increase the loss. So no scale above 12/7 of the base scale can be optimal. Lemma 4.2 then converts that ratio into a bound on the integer interpretation of the FP8 bit pattern, which is the move that makes this implementable: the sweep becomes an offset range over bit patterns rather than a search over reals, and the offset is at most +7.

The lower bound is where the paper is honest in a way worth noticing. For plain MSE, Lemma 4.3 proves the optimum is at least 4/5 of the base scale, giving a bit-pattern offset of -3. The proof needs a computer-assisted step: Table 5 lists convex certificates used to exclude the remaining FP8 ratios in the interval between 3/4 and 4/5, one ratio at a time. For the weighted objective, no such bound exists. The optimum can be arbitrarily small if the element carrying the block maximum happens to have near-zero importance. So they set the lower offset to -8 empirically and say so.

Eleven candidates for MSE, sixteen for WMSE, out of 126. That is the whole method.

3. The objective, and the part worth stealing

Minimising the error on the weight tensor is not the goal. Minimising the error on the layer output is. The paper writes the local reconstruction error as a trace over the Gram matrix of the input activations, notes that the full Gram matrix couples channels and is impractical, and takes the diagonal approximation that quantisation work has used since Optimal Brain Damage. The importance of input channel i becomes the squared L2 norm of that channel in the calibration activations.

That is standard. The extension is not.

For attention tensors there is no activation matrix to take norms of. So they route the importance from the operand on the other side of the multiply. Query states get their importance from the key cache. The key cache gets it from the query states. The value cache gets it from the output projection weights.

This is the most reusable idea in the paper and it occupies five lines of Section 4.1. Anyone quantising a KV cache under any format needs an importance signal, and this one is derivable from tensors already in memory.

4. The configurations, and the cost of each

Three settings, applied cumulatively. WA quantises weights and activations. WAKV adds the KV cache. WAKVQ adds the query states. Recovery is the average of five benchmark scores divided by the BF16 average.

Llama-3.1-8B-Instruct under GPTQ, which is the paper's own headline configuration:

The shape is consistent and it is the shape you would hope for: the harder the setting, the more the better initialisation is worth. At WA the gain over AbsMax is 1.69 points of recovery. At WAKVQ it is 3.46.

Across all four models, two backends and three settings, ScaleSweep beats AbsMax in 22 of 24 cells. It loses on Qwen3-8B at WAKV under GPTQ, where AbsMax reaches 97.29% against ScaleSweep's 96.04%, and on Qwen3-4B at WA under RTN by 0.08 points. The abstract says "consistently improves". The tables say generally, and the paper's own body text says generally too. We would take the body text.

The overhead is the easiest result to believe. Table 7 benchmarks Triton kernels against the default vLLM NVFP4 operator at hidden dimension 8192 on an RTX PRO 6000. The MSE variant runs at 0.97x to 1.08x the baseline, which is noise. The WMSE variant runs at 1.23x to 1.38x, settling at 1.37x for large batches. The operators are memory bound, so the extra arithmetic is nearly free. That is the same reason a GPU query engine wins or loses on bytes moved rather than instructions issued.

5. Where the gain actually came from

The paper's stated contribution is the bounds. Figure 3 suggests the bounds are not where the accuracy lives.

The figure sweeps ten bound configurations on Llama-3.1-8B-Instruct. The numbers match Table 3's GPTQ rows exactly, so the comparison is clean. Three readings matter.

The derived MSE range [-3, 7] gives 96.47 / 95.07 / 93.41 across WA, WAKV and WAKVQ. Truncating it to [0, 4], which discards most of what the theory bought, gives 96.52 / 94.59 / 92.34. At WA the truncated range is better. The authors say it plainly: "a narrower sweep range does not necessarily lead to worse performance and can occasionally achieve slightly better results."

Worse for the theory, the best WAKVQ number in the entire figure is 95.48%, from WMSE[-16, 7]. That is a range twice as wide as the one ScaleSweep ships, on the side where no bound was ever proved. It beats the reported 93.92% by 1.56 points of recovery.

Our position, which the authors do not state: the theory in this paper buys latency, not accuracy. The accuracy comes from sweeping at all instead of trusting a heuristic, and from weighting the error by channel importance. Neither is the bound. The bounds make the sweep cheap enough to ship, which is a real engineering contribution and the reason the 1.37x figure exists. But the paper is framed as a theory result with experiments attached, and the ablation reads better as an empirical result with a tractability proof attached. The authors half concede this when they write that "the local surrogate metric used in ScaleSweep cannot perfectly characterize the final downstream performance". That sentence is the most important one in the paper.

A second position: if we were deploying this, we would ship the MSE variant. WMSE wins on average, but it loses to plain MSE in 7 of 24 cells, including by 1.38 points on Llama-3.1-8B at WA under RTN and by 0.77 on Qwen3-4B at WA under GPTQ. It costs 1.37x on an operator that runs on every forward pass for activations, keys, values and queries. A free variant that is usually within a point of a taxed one is the better default, and the taxed one is worth it only once you have measured your own model in your own setting.

6. What the paper does not establish

The honest list is longer than the paper's own limitations section, which confines itself to noting that the results may not generalise to other low-precision formats.

Recovery is an average over five benchmarks, two of which barely move. HellaSwag and WinoGrande sit near saturation and quantisation hardly dents them: on Llama-3.1-8B at WAKVQ they retain 77.67 of 79.53 and 71.82 of 73.80. MMLU Pro in the same cell falls from 46.59 to 39.67. That is 85.1% retained on the benchmark that tests reasoning under knowledge load, inside a headline of 93.92%. Nothing is hidden, the full table is printed, but the summary statistic flatters the result and it is the summary statistic in the abstract.

Everything is simulated. The accuracy experiments run on NVIDIA L40 GPUs, which have no NVFP4 hardware. Only the latency microbenchmark touches Blackwell. No end-to-end throughput or memory figure is reported on hardware that executes the format natively, so the actual deployment case is unmeasured.

IFEval is unstable across the grid. On Llama-3.1-8B at WA under RTN, the MSE variant scores 73.20 and the WMSE variant 70.79, while BF16 scores 73.94. Elsewhere the ordering reverses. A 0-shot instruction-following score with 541 examples moving three points between two variants of the same method is within the range where the comparison does not discriminate, and several of the per-benchmark wins in Tables 3, 8 and 9 rest on it.

One calibration set, one seed. 128 sequences of 2048 tokens from RedPajama, seed 0, calibrated once. The one probe they do run is suggestive: swapping in the GSM8K training split lifts GSM8K by 2.88 points at WAKVQ while the five-benchmark average moves 0.44. Calibration distribution clearly matters and the sensitivity is not characterised.

No comparison against the methods that reshape the tensor. MR-GPTQ and the rotation-based family are discussed in related work and never measured against. ScaleSweep is compared only to AbsMax and 4/6, two initialisation heuristics it ought to beat, so its standing against the current best NVFP4 pipeline is unknown. We have made this point before in a reading of ViT-5: a component measured against one baseline is not measured. The experiment we want is whether a bounded scale sweep and a micro-rotation compose or compete. This paper does not run it.

None of this makes the paper weak. It makes it a scale-initialisation paper that should be read as one. The result we would carry forward is narrow and solid: under NVFP4, choosing the block scale by search rather than by heuristic is worth roughly 2 to 3.5 points of recovery at the aggressive end, and it costs nothing if you use the unweighted objective. That is the kind of gain that decides whether a model fits on the hardware you already have, which is the same arithmetic behind the economics of small models and the architecture choices large models make to stay servable.

7. Sources

1. Li Lin and Xiaojun Wan, ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization, Peking University, May 2026. Every table, figure, quotation and number above is from this paper.
2. Jack Cook, Junxian Guo, Guangxuan Xiao, Yujun Lin, Keith Wyss, Mahdi Nazemi, Asit Mishra, Carlo del Mundo, Tijmen Blankevoort and Song Han, Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling, the 4/6 baseline and the source of the global scale setting used here.
3. Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, one of the two PTQ frameworks ScaleSweep plugs into.
4. Yuxiang Chen and colleagues, TetraJet-v2: Accurate NVFP4 Training for Large Language Models with Oscillation Suppression and Outlier Control, for NVFP4 in training rather than post-training.
5. Mengzhao Chen and colleagues, INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats, on why format choice and granularity are not separable questions.
6. Our own GPU Query Engine for the memory-bound argument, and Mixture of experts: what actually changed for the other half of how large models are made servable.

What we did not verify: all of it, in the sense that matters. We have not run ScaleSweep, have not reproduced a single recovery number, and have no NVFP4 hardware to check the latency figures on. We read the lemma statements but not the appendix proofs in full, and we have not independently confirmed the computer-assisted exclusion in Table 5. The arithmetic we did do is the per-benchmark recovery in section 6, computed from the authors' own printed scores, and the win and loss counts across Tables 3, 8 and 9, which are ours and countable from the paper. Where we have taken a position the authors do not take, in section 5 and at the end of section 6, it is labelled and should be argued with.

This note sits in our Engineering track, alongside the foundation models and GPU acceleration threads. The full library is at Research.