Lynx: the KV cache does not have to arrive whole

This paper reads Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference by Wenchen Han at University College London, with Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste and Adam Barker at Huawei. The work and every figure below are theirs. What is ours is the reading: which result matters, where the gain actually comes from, and what the paper does not establish. We have not reproduced it.

Long-context serving has a cost nobody budgeted for. It is not attention, and it is not the model. It is the courier.

In a disaggregated deployment, prefill runs on one set of accelerators and decode on another. The split is a throughput win, because the two phases want different hardware. It also means the Key-Value cache has to cross a network, and until it lands the decode instance has nothing to attend over. That wait is pure latency, it lands in Time-to-First-Token, and the user feels all of it.

1. The arithmetic that makes transfer the constraint

The paper's worked example. Qwen3-235B-A22B, 94 layers, 4 KV heads, head dimension 128, one request at 128K context. In bfloat16 that is roughly 23.5 GB of KV cache for a single request. On a 100 Gbps TCP/IP interconnect, moving it takes about 2 seconds. Even on a 400 Gbps fabric, pulling long-context KV state out of remote storage costs over 0.5 second per request.

Compare that to what the decode step does with it. One token. The cache is enormous, the per-step compute against it is small, and the ratio worsens as context grows: cache size is linear in sequence length, while a decode step is one query against all of it. Same shape as the problem in our GPU query engine reference architecture. Once the kernels are fast, what decides your number is how far the bytes had to travel.

The existing fix is quantization: compress to INT8 or INT4 before it goes on the wire. That works, in the sense that half the bytes take half the time. The authors' objection is sharper than "it loses accuracy":

current KV compression schemes reduce how much data is transferred by not when decoding can begin.

Layer-wise pipelining is the usual answer, computing layer i while receiving layer i+1. It fails for exactly the case that matters. When communication time exceeds compute time, which is the regime the whole paper is about, decoding still stalls at every layer boundary. The barrier moved. It did not go away.

2. Not all bits are equal

The observation the system is built on: within a quantized KV value, the most significant bits set the magnitude of attention scores and largely preserve the ranking of token associations, while the least significant bits refine precision. Attention is a softmax over scores, so ranking is most of what it needs. If that holds, the cache is not an indivisible unit. Start on the coarse structure, fix it up when the detail arrives.

The catch is that a naive MSB split off a standard quantization grid does not give you coarse structure. It gives you noise, because of outliers. A small subset of KV channels holds values up to 100 times larger than the rest, and the paper derives the consequence precisely: effective bin count for a non-outlier channel is 2^B · m_i / m, so at a 100-times ratio an 8-bit grid with 256 bins collapses to roughly 1.5 bits of usable resolution for most channels. Cut the top four bits off that and you have cut four bits off almost nothing.

Removing the outliers does not rescue it either. After per-channel normalisation the remaining values are still clustered near zero in a Laplacian-like shape: 50 per cent of KV elements sit inside the middle 17 per cent of the value range. A linear grid spends its bins on empty tails and starves the dense centre. So the split requires a quantizer built for it, and that is the part of the paper doing the real work.

3. The quantizer

Four stages, per page of 256 tokens, which keeps the scheme compatible with the paged attention layout serving engines already use.

Page-level normalisation. Per channel, take the page minimum and range, map to [0,1].

Chunk-wise outlier isolation. Subdivide each channel's page into chunks of C = 32 consecutive tokens, then centre by a local mean and divide by a local max-absolute scalar. The point is containment: one extreme value inflates the scale of its own 32-entry chunk rather than setting the step size for the whole page. Outliers concentrate in a few channels and are of consistent magnitude within a channel, so per-channel, per-chunk is the granularity that matches the data.

Non-linear transform. An alpha-law logarithmic map onto 2^7 - 1 = 127 levels, so bins follow the distribution rather than the range.

The cut. This is the elegant part. After a logarithmic transform the integer behaves like a float. The top 4 bits partition the dynamic range into geometric intervals, so reconstructed magnitude scales as (1+alpha)^q: an exponent. The bottom 4 bits linearly interpolate inside the interval the anchor named: a mantissa. The anchor stream alone carries order of magnitude, which is what an attention score needs to rank correctly.

Two details that look small and are not. A round-half-down bias of +7 applied before the shift makes the residual a centred signed correction rather than a one-sided one. And a sign-aware mapping sends negative zero to distinct negative integers, preserving the sign of small-magnitude values. Lose the sign in the low-precision regime and the attention score points the wrong way, which no residual will fix in a prefix already drafted.

Total budget: 4 bits anchor plus 4 bits residual, an effective INT8 representation. Not a compression win over INT8. A scheduling win.

4. Speculation at the communication boundary

Draft-then-verify, generalised somewhere it has not been applied before. In standard speculative decoding the draft comes from a smaller model. Here there is no second model: the approximation is the target model running on its own truncated cache.

Anchor stream lands, decoding starts and marks its tokens as drafts. The residual stream transfers concurrently under strict stream-level priority, anchor buffers always drained first. When the residual completes and the kernel refines the cache to Q_full, the decoder runs one parallel forward pass, accepts the longest matching prefix, and corrects the first divergence without recomputation.

The acceptance numbers decide whether this pays, and they are good. On MMLU-Pro with Qwen 32B, Lynx generates 21.43 speculative tokens on average and accepts 19.38. The whole draft survives 64.8 per cent of the time. Over a long proposal run there is an 88 per cent chance at least 10 tokens are accepted and a 70 per cent chance at least 20. Speculation is capped at 64 tokens because acceptance thins past that, after which the decoder waits.

Acceptance that high is worth sitting with. A draft model has to be trained to agree with its target. The anchor stream agrees because it is the target, reading a blurred copy of the same cache. That is a structurally better draft source than any distilled small model, and the paper does not say so.

5. The numbers

Qwen 32B, MMLU-Pro, 16K context, from the paper's headline table.

That is INT4's start time at BF16's quality. 1.43 times better TTFT than INT8, a 30.5 per cent reduction, and 10.7 per cent off TT32T. The latency section states the same comparison as 0.87s faster to first token and 0.39s faster to the 32nd, roughly 29 and 13 tokens of output time. Note that the table's own figures give a 0.7s TTFT gap, not 0.87s, so the two reports are of slightly different configurations or the text is loose. The extra compute Lynx spends on quantization, decompression and rejected drafts is 0.13s here, and the reported latencies already include it.

The reconstruction error table is the more interesting one. Normalised mean squared error of attention output, LLaMA 8B on MMLU:

Lynx's full reconstruction is about 25 times more faithful than uniform INT8 at the same 8 bits per coordinate, and Lynx-INT4 is 35 times more faithful than uniform INT4 at the same 4 bits. End to end on MMLU with Qwen, INT4 drops accuracy 8.7 per cent against Lynx-INT4's 1.7; on Needle with Mistral the ROUGE-L drops are 4.5 and 0.9.

Scaling behaves the way the mechanism predicts. The TT64T advantage over INT8 grows from 0.22s at 32K context to 0.84s at 128K, because a longer residual transfer buys more speculative tokens, and shrinks from 0.86s at 10 Gbps to 0.18s at 50 Gbps, because a faster link leaves less to hide behind.

6. Where the gain actually comes from

This is one paper containing two contributions, and the variants let you separate them: Lynx-INT8 and Lynx-INT4 are the hierarchical quantizer without split-stream transfer.

Read that way, all of the accuracy comes from the quantizer and all of the latency comes from the split-stream. Lynx and Lynx-INT8 are indistinguishable on quality, inside the ±0.3 per cent the authors call random variation, and they send the same total bits, so the entire TTFT gap between them is scheduling.

Our position: the quantizer is the more durable half, and the title bills it as a means to the transfer win rather than a result in its own right. A 25-times error reduction at a fixed bit budget is worth having with no network involved at all. It applies to KV offloaded to host DDR, to cache held on NVMe, to anything that pays to store the cache smaller. The split-stream trick has a value strictly proportional to how slow your interconnect is, and the bandwidth sweep shows that window closing: a 0.18s win at 50 Gbps is also available from a procurement team. Progressive execution is the better story. Non-linear chunk-wise quantization is the better product.

A second position, on what verification actually guarantees. The abstract offers "verification that ensures equivalence to higher-precision decoding". Verification guarantees that output follows the distribution of the target model as reconstructed from Q_full, and Q_full is Lynx's own effective-INT8 reconstruction, not BF16. The implementation section says so plainly: the protocol ensures "the same accuracy as decoding with the higher precision (i.e., INT8) KV cache." The BF16-equivalence claim in the abstract is empirical, resting on INT8 being close enough to BF16 to start with, and the vNMSE of 0.00017 is evidence for that rather than proof of it. The formal guarantee and the headline claim are not the same claim. We made a version of this argument about what happens when the verifier moves inside the model: a correctness check is only as strong as its reference, so name the reference out loud.

7. What the paper does not establish

No GPU results. The prototype is roughly 2k lines of Ascend-C kernels and 2k lines of Python on Ascend 910B4 NPUs, integrated into vLLM-Ascend with LMCache-Ascend as the connector. Portability to NVIDIA is asserted and left to future work. The anchor path depends on bit-packing and memory-coalesced kernels, so it is a reasonable claim and an unmeasured one.

No throughput number anywhere. Every latency figure is Time-to-kth-Token, a single-request metric. Speculative decoding on the anchor stream consumes decode-instance compute, and in production that instance is serving other requests. This is the most important missing experiment: under batch pressure the drafts are not free, they are paid for in capacity that would have served someone else, and nothing here says where the break-even sits.

Bandwidth is emulated. Inter-server bandwidth is rate-limited from 10 to 50 Gbps to recreate the transfer-to-compute ratio of a million-token workload inside a testbed of two 8-NPU servers. The stated equivalence, 128K at 10 Gbps standing in for 1M at 80 Gbps, is a ratio argument. Reasonable, and not the same as running the workload.

One bit-width configuration. Only 4-plus-4 is evaluated. The authors are explicit that INT6 or INT2 splits are future work, which leaves the quality-latency frontier unmapped.

No component ablation inside the quantizer. Page normalisation, chunk-wise outlier isolation, the logarithmic transform, the round-half-down bias and the sign-aware mapping are evaluated only as a bundle. Lynx-INT4 against INT4 shows the bundle is worth 35 times in vNMSE. Which stage earned it is unknown, and chunk size is fixed at C = 32 with the claim that smaller chunks reduce error and no sweep to show it. Same gap we flagged in Kimi K3: a recipe measured against a baseline tells you the recipe works, not which ingredient did it.

The CacheGen comparison is a self-port. No official Ascend implementation exists, so the authors wrote their own and report only its accuracy, not its latency. The restraint is correct, since compression fidelity is hardware-independent in a way timing is not. The 5.1 per cent accuracy headline is still measured against a baseline they built themselves.

The long-context comparison runs in a hard regime. At 128K on LLaMA 8B the accuracy axis in the paper's own figure sits in the low thirties for every method. Degradation rankings matter less when all candidates are that far down, and the relative drops there are reported without absolute figures in the text.

8. Sources

1. Wenchen Han, Gingfung Matthew Yeung, Marco Barletta, William Toner, Amory Hoste and Adam Barker, Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference. University College London and Huawei. Every figure, table and quotation above is from this paper.
2. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre and John Jumper, Accelerating Large Language Model Decoding with Speculative Sampling, the draft-then-verify protocol Lynx moves to the communication boundary.
3. Yuhan Liu et al., CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving, the compression baseline in section 5.
4. Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer and Amir Gholami, KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization, representative of the KV quantization line Lynx positions against.
5. Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer, LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, where the activation outlier problem of section 2 was characterised.
6. Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, the paged cache layout Lynx quantizes in page units to stay compatible with.
7. Our own Attention Is All You Need, nine years on, for why there is a KV cache to move in the first place.
8. Our own Memory leaves the model, on the broader pattern of inference state being pulled out from under the model.

We have not run Lynx, do not have Ascend hardware, and have reproduced none of these measurements. Everything above is a reading of a published paper and the figures are the authors'. We did not verify the vNMSE computation, the 23.5 GB cache calculation beyond checking it against the dimensions the paper states, or the claim that the design ports to GPUs. Where we have taken a position the authors do not state, which is most of sections 6 and the throughput argument in section 7, it is our argument and should be argued with.

This note sits in our Engineering track, alongside the foundation models and GPU acceleration threads. The full library is at Research.