In December 2023, Mistral shipped Mixtral 8x7B: eight experts, two active per token, 46.7 billion parameters of which 12.9 billion ran. It was the model that made mixture of experts feel normal.
In August 2026, Moonshot shipped Kimi K3: 896 experts, sixteen active, 2.8 trillion parameters of which 104 billion run.
The expert count went up by a factor of 112 in under three years. That is the number people quote. It is also the least interesting thing that happened, because the architecture did not just get bigger. It changed shape three times, and each change is a decision a working engineer can recognise.
1. The arc, in one table
| Mixtral 8x7B | DeepSeek-V3 | Kimi K3 | |
|---|---|---|---|
| Released | Dec 2023 | Dec 2024 | Aug 2026 |
| Routed experts | 8 | 256 | 896 |
| Active per token | 2 | 8 | 16 |
| Shared experts | 0 | 1 | 2 |
| Total parameters | 46.7B | 671B | 2,800B |
| Active parameters | 12.9B | 37B | 104B |
| Share active | 27.6% | 5.5% | 3.7% |
Read the last row rather than the first. The proportion of the model that runs for any given token fell from roughly a quarter to roughly a twenty-fifth. Mixtral ran one parameter in 3.6. Kimi K3 runs one in 27.
That is the actual trend. Everything below is how it was made to work, because at eight experts none of it was necessary and at 896 all of it is.
2. Experts got small enough to combine
The first change is the one the rest depends on, and it is almost embarrassingly simple: split the feed-forward block into more, smaller pieces.
At eight experts, routing is a choice between specialists. A token goes to the two that suit it. The granularity is coarse, so "suits it" is a blunt judgement, and most of what a token needs is probably in an expert it did not pick.
At 896 experts with sixteen active, routing is a composition. The model is not picking the right specialist, it is assembling a combination, and the number of distinguishable combinations is astronomically larger than the number of experts. That is where the capacity actually comes from, and it is why expert count grew faster than active count.
The survey work calls this expert granularity and treats it as the first of several independent axes rather than a tuning knob. Jiguo Li's 2026 survey organises the whole lineage along five of them, and its framing is the sharpest one-line summary of the field we have read: the shift is "from merely activating more sparse parameters toward decoupling semantic routing, computational budgets, and physical execution".
Those three used to be the same decision. Now they are three.
3. Something has to hold what everything needs
Fine-grained experts create a problem immediately, and it is a problem anyone who has decomposed a system too eagerly will recognise.
If every expert is a specialist, who holds the common knowledge? Grammar, basic syntax, the transformations every token needs regardless of topic. Either each specialist relearns it, which duplicates the same weights hundreds of times and wastes most of the pool, or the router sends every token to the same few experts for the common part, which is load imbalance by construction.
The answer is the shared expert: one or more experts that run for every token, unrouted, alongside whichever specialists the router picks. DeepSeek's work introduced them to mitigate exactly this knowledge redundancy. DeepSeek-V3 runs one. Kimi K3 runs two, full-width.
It is a small structural change and it is the right one. The specialists get to be specialists because something else is carrying the floor. We have made the same argument about systems rather than models in a chain of events beats one big model: specialisation works when the recurring common case is handled somewhere that is not a specialist.
4. Balancing moved out of the loss
This is the change we would point a systems engineer at, because the lesson generalises past machine learning entirely.
A router learns. Learned routers concentrate: a few experts become popular, get more gradient, become better, get picked more. Left alone, a 896-expert pool collapses into the forty that always fire, and you have paid to store 896 and bought the behaviour of forty.
The original fix was an auxiliary loss: add a term penalising imbalance. It works, and it has an obvious cost that everyone accepted for years. The model is now optimising two objectives at once, and the balancing term is in direct tension with the quality term. You are explicitly trading away capability to keep the load even.
DeepSeek-V3 removed the trade. Instead of a loss term, it keeps a per-expert bias that is added to the affinity score when the router picks, and adjusts it by hand: decrease the bias by a step when an expert is overloaded, increase it when underloaded.
The detail that makes it work is worth stating precisely, because it is the whole trick:
the bias term is only used for routing. The gating value is still derived from the original affinity score.
So the bias changes which experts are chosen and never how much their output counts. It sits outside the gradient. The loss is left alone to optimise quality, and balance is maintained by a controller beside it rather than a penalty inside it.
Kimi K3 does its own version, which Moonshot call Quantile Balancing, and names the same reason: the problem worsens as the pool grows. No mathematics is published for it.
The general form of this is a good engineering instinct and it is not specific to models. When two objectives are in tension, look for a way to move one of them out of the optimiser and into a controller. You usually cannot. When you can, you stop paying for the compromise.
5. What it costs, which is the part that gets skipped
DeepSeek-V3's full training run was 2.788 million H800 GPU hours, which the report prices at 5.576 million dollars at two dollars an hour, over 14.8 trillion tokens.
For a 671-billion-parameter frontier model that is a famously small number, and the sparsity is most of the reason: you train and run 37 billion parameters per token, not 671. But read it the other way as well. Five and a half million dollars is roughly 150,000 dollars per billion active parameters, and that is the floor for a lab that already has the cluster, the data and the recipe.
The economics are the point of the architecture, not a side effect of it. We priced where that lands for a buyer in what the frontier costs now, and the serving side of the same question in Kimi K3: where the compute went.
6. What is still unresolved
Three things, stated as open rather than solved.
Nobody publishes ablations at the frontier. Moonshot are explicit that their 2.5x scaling efficiency claim compares two complete recipes and that no single change can be isolated. That is honest and it is also a wall: the field cannot say how much of the gain is granularity, how much is shared experts, how much is balancing, and how much is data.
Routing is still learned, and learned routing still concentrates. Quantile Balancing and bias controllers manage the symptom well. Neither explains why a router concentrates in the first place, and the problem is stated as getting worse with scale, which means the current answers are holding rather than solving.
Serving a sparse model is harder than training one. 3.7 per cent active is a wonderful number for FLOPs and a difficult one for memory: all 2.8 trillion parameters have to be somewhere reachable, which is why we worked out in the token cost audit that self-hosting almost never pays. Sparsity moves cost from compute to memory and interconnect. It does not remove it.
7. What a team building systems should take from this
Not "use a mixture of experts". Almost nobody reading this is training one.
Granularity changes the kind of decision you are making. Coarse pieces force a choice between them; fine pieces let you compose. If routing in your system feels like picking the least-wrong option, the pieces are probably too big.
Specialisation needs a floor. Decompose a system into specialists and something still has to own the common path, or every specialist reimplements it. Name that thing deliberately.
Move a competing objective out of the optimiser when you can. The auxiliary-loss to bias-controller shift is the cleanest example we have seen recently of a trade that looked inherent and was not.
8. Sources
- Jiguo Li, The Evolution of Mixture-of-Experts Architectures in Large Language Models, 2026. The five-axis framing in section 2 and the quoted line about decoupling routing, budgets and execution are from this survey.
- DeepSeek-AI, DeepSeek-V3 Technical Report. Every DeepSeek-V3 figure above, the auxiliary-loss-free bias mechanism and the quotation in section 4, and the training cost in section 5.
- DeepSeek-AI, Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts, 2024, the paper the bias controller comes from.
- Moonshot AI, Kimi K3: open frontier intelligence, for the 896-expert configuration and Quantile Balancing, read at length in where the compute went.
- Mistral AI, Mixtral of Experts, 2024, for the 8x7B configuration.
- Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, 2017, where the modern form of this starts.
All three architectures above are other people's work and the figures are theirs. The arc drawn through them, the sparsity reading in section 1 and the engineering lessons in section 7 are ours, and should be argued with. We have not trained a mixture of experts model and make no claim to have.
This note sits in our Research track, alongside the foundation models and system design threads. The full library is at Research.