Moonshot AI published the Kimi K3 technical report on 7 August 2026, alongside open weights on Hugging Face. The model is 2.8 trillion parameters in total with 104 billion active per token, native vision, and a one million token context window.
We have already priced it. In Kimi K3 vs Claude: what tokens actually cost we worked out that it sits above Claude Sonnet 5 on the meter, that self hosting 1.4 terabytes of weights almost never pays, and that the received wisdom about cheap Chinese models stops being true at the frontier tier.
This note is about the other half of that, which is why the model costs what it costs. The report is unusually clear that the architecture was chosen to move a curve rather than to win a table, and the headline claim is a compute claim, not a quality one: roughly 2.5 times more validation accuracy per unit of training compute than Kimi K2.
1. The 2.5 times claim, and what it is not
Start with the caveat, because Moonshot put it in themselves and it is to their credit.
The figure comes from fitting scaling law curves for K2 and K3 on a plot of log training FLOPs against validation loss, and measuring the horizontal distance between them at matched loss. That distance is about 2.5 times. Read correctly, it says: to reach the same validation loss, the K3 recipe needs about 2.5 times less training compute than the K2 recipe.
What it does not say is which change bought it. The report is explicit: "because the comparison is always the full K3 versus K2 recipe, it is not possible to isolate how much any single change contributed". There are no component ablations. The attention change, the mixture of experts change, the cross-depth residuals and the data recipe are measured together, as one thing.
We would rather have the ablations. But a system-level comparison honestly labelled as one is worth more than a component number quoted without its confound, and the shape of the claim tells you what the team was optimising. Nobody fits a scaling curve against the previous generation unless the question they were asked was how far the same budget goes.
2. Bounded state in three layers out of four
The attention scheme is the change with the most downstream consequence, and it is the one to understand first.
Standard attention stores a representation for every token seen so far, and consults all of them for every new token. That is what makes it work, and it is also what makes long contexts expensive: the cache grows linearly with the conversation and you carry it in memory for the whole session. This is the trade we traced back to its origin in Attention Is All You Need, nine years on, and it has been the defining cost of the architecture for most of a decade.
Kimi Delta Attention does something different in three layers out of every four. Instead of keeping every token, it keeps "a fixed-size summary of past context, a small matrix updated at each token step". The state does not grow. A KDA layer consulting a million tokens of history holds exactly as much as it holds at ten thousand.
Every fourth layer is conventional global attention, which Moonshot call Gated MLA, "to let the model make unrestricted connections across the entire input". That ratio is the design. Bounded state almost everywhere, with a regular opportunity to reach back at anything without compression.
There is a detail in the report that is worth pulling out because it shows the thinking. KDA bounds the decay rate it uses to downweight older information, and the reason given is not about quality. It is that bounding the decay "keeps internal numerical values within a range that GPU hardware can handle efficiently with its fastest matrix-multiply units, eliminating a slower fallback computation path".
That is a mathematical choice made to stay on a hardware fast path. It is the same species of decision as everything we wrote about in the GPU query engine: at this scale the algorithm and the silicon are not separable concerns, and a formulation that is slightly worse on paper but stays in the fast kernel wins by a margin no amount of cleverness recovers.
The report publishes no KV cache footprint, no tokens per second and no decode latency. We would like all three. What the mechanism implies is clear enough: the memory you carry for a long session is dominated by the one layer in four that keeps everything, not by all of them, which is what makes a million token window a product decision rather than a research demo.
The third architectural piece is Attention Residuals, which let each layer "look back at the outputs of all earlier groups of layers, weighting them based on how relevant each group is for the current step". Residual connections have always let information skip forward a layer. This makes skipping forward selective and long range, across the depth of the network rather than along the sequence. No isolated ablation is given for it either.
3. Sparsity, and the number that actually matters
2.8 trillion parameters is the number that gets quoted. 104 billion is the number that decides what it costs to run.
The model holds 896 experts. Each token activates 16 of them, plus two full-width shared experts that run for every token and absorb the transformations common to everything. That is 3.7 per cent of the parameters doing work at any moment.
Two details in how the routing is done are worth more than the headline counts.
The routing itself "happens in a compact lower-dimensional space". Choosing between 896 options is not free, and at 896 the choosing becomes a cost of its own if you do it in the full hidden dimension. Compressing first makes the decision cheap enough to not matter.
And Quantile Balancing exists because "no expert sits idle while others are overloaded, a problem that worsens as the expert pool grows". This is the failure mode of large expert pools and it is worth being concrete about. Routing is learned, learned routing concentrates, and a concentrated router gives you a model that is enormous on disk and small in effect: you pay to store 896 experts and get the behaviour of the forty that ever fire. Every scaling story about expert count is really a story about whether the balancing held.
No mathematics is given for Quantile Balancing and no comparison against standard load-balancing losses. It is named, not derived.
What the shape means in practice is the thing we worked through in small language models: the agent economics. The industry keeps discovering the same result from different directions: most of what a model does on any given token does not need most of the model. A sparse mixture is one answer to that. Running a small tuned model for a known job is another, and the two are not in competition, they are the same observation applied at different granularities.
4. Nine specialists, collapsed back into one
The post-training recipe is the part of this report we find most interesting, because it is an argument we have been making ourselves arriving from inside a frontier lab.
Reinforcement learning is run across three task families, which are general reasoning and knowledge work, software engineering, and autonomous agent tasks, at three effort levels from fast to exhaustive. Three by three gives nine specialist models.
Then the nine are collapsed. A single model is trained to imitate all nine simultaneously, weighted by domain and effort level, and that merged model is what ships.
So the specialisation is real and it happens during training. The reason it does not ship as nine models is operational, and it is the same reason we gave against a fleet of specialists in a chain of events beats one big model: every model you ship is a thing to version, serve, evaluate and keep from drifting, and nine of them is nine of each. Moonshot pay that cost once, in the training pipeline, and hand the user one artefact.
It is worth being precise about what this does and does not settle. It does not show that a general model beats specialists. It shows that specialists are an efficient way to generate training signal, and that the signal survives being merged. Whether a merged generalist matches nine served specialists at inference time is a different question and the report does not answer it.
The effort axis is the other half, and it is the same idea we found in when the verifier moves inside the model: how hard to think is being treated as a trained dimension rather than a runtime setting bolted on afterwards. A model that was trained at three effort levels has learned what a cheap answer and an exhaustive one look like for the same question. That is a more coherent thing than a temperature knob.
And the fact that any of this is affordable at all is downstream of what we wrote about in Tinker, Inkling and the return of fine tuning. Specialising a model used to cost as much as a project. Nine of them inside one pipeline is a sentence that would not have been written three years ago.
5. The unglamorous part that unlocks the rest
Buried in the infrastructure section is the piece of engineering we would most want to steal.
Training an agent on long-horizon tasks at million token scale means episodes that run for a very long time and frequently hit timeouts. The obvious behaviour is to discard and restart, which wastes everything the episode had built. Moonshot instead save the sandbox environment, "a virtual computer's complete state", and resume it later.
The detail that makes it work: "the model's internal summary states from KDA are saved alongside the conventional attention cache, so nothing needs to be recomputed when a paused session continues". The bounded KDA state is small enough to checkpoint. The architecture choice from section 2 is what makes the infrastructure choice in section 5 possible, which is the kind of coupling you only find by building the whole thing.
The result is that checkpoint and resume are "fast enough to be practical inside a training loop". No figures are given for checkpoint latency or state size.
We would make the general version of this claim loudly. The capability ceiling on long-horizon agent work is not usually model quality. It is whether your harness can survive an episode being interrupted. If a timeout costs you the episode, your maximum task length is set by your least reliable dependency, and no amount of model improvement raises it. Resumable state is the thing that moves the ceiling, and it is ordinary systems engineering that nobody writes a launch post about.
The same principle is why we argued in AI implementation is an architecture decision that the model is rarely the constraint. It is not the constraint inside a frontier lab's training loop either.
6. Where it actually lands
Moonshot's own summary is more honest than most. K3 "still trails the most powerful proprietary models, namely Claude Fable 5 and GPT-5.6 Sol". It "leads or ties on several coding and agent tasks while trailing on graduate-level research reasoning, knowledge-work Elo comparisons, and harder computer-use benchmarks".
The headline result is BrowseComp at 91.2 per cent, against 90.4 for GPT-5.6 Sol and 88.0 for Claude Fable 5, with all figures at maximum reasoning effort. It also leads on ProgramBench and SWE-Marathon.
Note what those three have in common. BrowseComp is long multi-step retrieval. SWE-Marathon is long software engineering. These are the benchmarks where the thing being measured is how long a model can keep going without losing the thread, and that is exactly what this architecture was built for: bounded memory so long contexts stay affordable, a merged agent specialist in the post-training mix, and a training harness that could actually run episodes that long.
The gaps are equally consistent. Graduate-level research reasoning and knowledge-work Elo are depth-per-token measures, and nothing in this report targets depth per token. An architecture optimised for accuracy per unit of compute is not the same as one optimised for the hardest single answer, and K3 lands exactly where its own design says it should.
That is the most useful thing a technical report can do: make the result predictable from the method. We should all be more suspicious of reports where it does not.
7. What a team should take from it
Three transferable things, none of which require 2.8 trillion parameters.
Bounded state beats unbounded state wherever you can tolerate it. The lesson of KDA is not the specific mechanism, it is the ratio. Most of your system does not need full history, some of it does, and the engineering win is in being deliberate about which is which rather than giving everything the expensive option by default. We made the application-level version of this case in read and write memory.
Specialise to generate signal, consolidate to ship. Nine training specialists and one served model is a pattern worth copying at any scale. The thing you tune against and the thing you deploy do not have to be the same artefact.
Make your harness resumable before you make your agent smarter. If an interrupted episode costs you the whole episode, your task length ceiling is set by your flakiest dependency, and it does not move when the model improves.
And one caution. This is a technical report, not a paper with ablations. The architecture claims are plausible, internally consistent and unverified from outside. The weights are published, so unlike most frontier releases somebody can check. Until somebody does, every number above is Moonshot's.
8. Sources
- Moonshot AI, Kimi K3: open frontier intelligence, submitted 7 August 2026. Every architectural detail, quotation and benchmark figure above is from this report. The work is Moonshot AI's; this note is our reading of it.
- Kimi K3 weights on Hugging Face, published by Moonshot AI so the claims above can be checked independently.
- Moonshot AI, the lab, for deployment notes including memory requirements and known behavioural limitations.
- Vaswani et al, Attention Is All You Need, the mechanism Kimi Delta Attention is a departure from, read at length in our nine years on note.
- Our own token cost audit of Kimi K3, which covers pricing, self hosting economics and the strategic context this note leaves out.
We have not run Kimi K3 at scale and we have not reproduced any figure in the report. Where we have drawn a conclusion the report does not state, which is most of sections 5, 6 and 7, it is our argument and should be argued with.
This note sits in our Research track, alongside the foundation models and system design threads. The full library is at Research.