Small language models: the agent economics

Watch what an agent actually does over a working day and the shape of the work is not what the marketing suggests. It classifies an email. It pulls three fields out of a PDF. It decides which of four routes a request takes. It formats a response into a schema. It calls a tool and checks whether the result looks wrong.

Those are narrow, repetitive, tightly specified jobs. Perhaps five percent of the calls involve anything you would recognise as reasoning. Yet almost every system we are asked to review sends all of them to the same frontier model, because that is the model somebody picked at the start and nobody revisited.

NVIDIA put the argument formally in a 2025 position paper, Small Language Models are the Future of Agentic AI: small models are sufficiently powerful for the bulk of agent invocations, inherently more suitable for them operationally, and necessarily more economical. We think that is broadly right, with one important caveat about how you measure it, and we have watched the evidence accumulate from an unexpected direction.

1. What counts as small, and why the line sits where it does

There is no formal definition. The working threshold in the research is roughly ten billion parameters, and the practical test is better: can you serve it on one accelerator, without a multi node deployment. That is the boundary where the operational character of the thing changes. Below it you can run the model next to your data, own the latency, and stop paying per token. Above it you are running infrastructure.

The examples the NVIDIA paper cites are all in that band, Microsoft's Phi 3 small, NVIDIA's own Nemotron H, HuggingFace's SmolLM2, and the DeepSeek R1 distillations. What they have in common is not architecture. It is that each was built to be good at a bounded set of things rather than at everything.

Three techniques do most of the work: distillation, where a small model learns from a larger teacher; quantisation, where 32 bit weights come down to 8 or 4 bits with modest quality loss; and pruning, which removes connections that were not earning their keep. None is new. What changed is that they now compose well enough to land a genuinely useful model inside a single card.

2. The evidence arrived from an odd direction

The most convincing datapoint we have written up did not come from the language model world at all.

Google's TimesFM 3, released on 31 August 2026, is 330 million parameters and takes the top rank on three forecasting benchmarks ahead of a family that runs up to 2.5 billion. Roughly eight times smaller than the largest model it beats. We went through the architecture in the TimesFM 3 note, and the relevant part here is that it wins by being the right shape for a bounded problem, not by being scaled harder at a general one.

The counter example is just as instructive. Kimi K3 is 2.8 trillion parameters with open weights, and when we worked through the self hosting maths the break even against the vendor's own API sat somewhere between 5.7 and 12.8 billion tokens a month. Open weights on a model that size buy auditability and portability. They do not buy a lower bill. Open weights on a model small enough to serve on one card buy both, and that difference is the whole argument for the category.

You can see the same pattern in our own product work. The embedding and scoring layers in the transaction foundation model were never going to be frontier sized, because the task is narrow and the volume is enormous, which is exactly the profile that rewards a specialist.

3. The caveat: cost per token is the wrong metric

Here is where the small model case gets oversold, and where we part company with the more enthusiastic version of it.

A model that is ten times cheaper per call and needs three extra turns to get the same result has not saved anything. Add one human correction and it has cost more, because the expensive resource in most businesses is not tokens, it is people re doing work the system got wrong. The number that settles it is cost per completed task: turns to completion, retry rate, and correction rate, measured on your real traffic.

We made the same point about model tiers in the Kimi K3 note and it holds here with more force, because the quality gap between a small specialist and a frontier model on a task the specialist was not trained for is much wider than any price table suggests. The failure mode is specific and recognisable: the small model is fine on 90 percent of cases and confidently wrong on the tail, and the tail is where the money is.

Which is why the routing decision in the flow above turns on the nature of the work rather than on the importance of the customer. Novel reasoning, genuine ambiguity, and anything the task specification does not cover go to the frontier model. Everything with a known shape goes to the specialist. Both return through the same output contract, so the rest of the system does not know or care which answered, and so the routing stays reversible when you get it wrong.

4. Where the small model comes from

You mostly do not go shopping for it. You distil it from work you have already done.

If your agent has been running against a capable model, you are sitting on the training set: logged trajectories of real inputs and good outputs on the exact distribution you care about. That is a far better corpus for your task than any general benchmark, and it is a byproduct of operating the system rather than a separate procurement exercise.

The gate before training is coverage, not volume. Ten thousand examples of the easy case teach a model to be confident about the easy case. What you need is the awkward tail: the malformed input, the ambiguous category, the supplier who formats invoices differently. If those are not in the log, collect more before you train, because a specialist trained without them will fail precisely where the frontier model was earning its price.

Then score honestly, on held out examples, against the model you are replacing, with a metric somebody in the business actually cares about. Ship it behind the routing layer so you can pull it back in an afternoon.

One thing worth saying plainly: this is a governance improvement as well as a cost one. A narrow model doing one job with a scoped credential is easier to reason about than a general model with broad tool access, and it makes the approval lanes in Governed autonomy cheaper to draw. It does not, however, touch prompt injection, credential scope or supply chain risk, which are properties of the agent architecture rather than of model size. We set those out in the agent security paper, and a small model changes none of them.

5. What to do about it

Five steps, and the first is a measurement rather than a migration.

1. Log what your agent is actually doing, broken down by step type. Most teams are surprised by the histogram: the overwhelming majority of calls are formatting, extraction, classification and routing.
2. Pick the highest volume boring step and nothing else. One step, the most repetitive one you have.
3. Put a routing layer in first, before you have anywhere to route to. It is a half day of work, it is reversible, and without it every later experiment needs a deployment.
4. Try an off the shelf small model on that step before you train anything. Often it is enough, and distillation is a real project you should only start once you know it is needed.
5. Score cost per completed task, not per token, and include the correction rate.

The broader point is the one running through most of what we publish. The frontier is genuinely remarkable, and the interesting engineering question is almost never how to use more of it. It is which parts of a system need it at all. As we argued in Brain blueprints of AI, capability and appropriateness are different axes, and systems get good when you stop confusing them.

If you want the routing layer and the measurement built properly, that is a short engagement with an unusually clear payback. Talk to us, or see how we build systems for businesses.

6. Sources

1. Belcak et al, NVIDIA Research, Small Language Models are the Future of Agentic AI, arXiv 2506.02153, the position paper this note argues with and mostly agrees with.
2. NVIDIA Research, the accompanying correspondence page, which collects responses and counterarguments.
3. Google Research, TimesFM 3, for the 330 million parameter result discussed above.
4. HuggingFace SmolLM2 and Microsoft Phi, as current examples of the category.

Benchmark claims about specific small models beating specific large ones are usually true on the narrow task they were measured on and rarely transfer. Treat any such figure, including the ones quoted here, as a reason to run your own evaluation rather than as a result you can adopt.

This note sits in our Research track, alongside the foundation models, agentic and operations threads. The full library is at Research.