Kimi K3.1 Pro: one benchmark, three numbers

This paper reads the launch of Kimi K3.1 Pro by Moonshot AI, released on 23 October 2026, together with the independent measurements published by Artificial Analysis the same day. The model, the training and every benchmark figure attributed to Moonshot are theirs. The third-party scores are Artificial Analysis's. What is ours is the reading: which number is load bearing, what each one is a number of, and why three of them disagree. We have not trained, served or reproduced this model.

Kimi K3.1 Pro is the strongest open-weights-to-be model yet measured against the frontier. It is also the clearest case this year of a launch benchmark that cannot be read as a ranking, because on its headline coding benchmark the model has three published scores from its own launch week, and the spread between them is wider than the gap between the top five models on that leaderboard.

Both of those things are true, and the second is the more useful one.

1. What actually shipped

Moonshot describes K3.1 Pro as "a post-trained version of Kimi K3, with substantially larger post-training compute." The base model is unchanged, which matters: this is not a new architecture, and the gains are a post-training result rather than an architectural one. We read the K3 architecture itself in Kimi K3: where the compute went, and nothing in that reading needs revising.

A second variant, kimi-k3.1-pro-highspeed, runs roughly two to three times faster at twice the rate, 4 dollars in and 20 out, with cached input priced the same as the standard model.

The 104B active against 2.8T total is a sparsity ratio of about 1 in 27, which is where the frontier has been converging all year for reasons we set out in what actually changed in mixture of experts. It is not the story of this release.

The weights are the part that makes this launch structurally different from the proprietary ones. Artificial Analysis puts K3.1 Pro at 55 on its Intelligence Index, fourth overall, behind Claude Opus 5.5 at 58 and Claude Sonnet 5.5 and Muse Wonder Full at 56. Kimi K3 scored 44, so this is an 11 point jump in one post-training cycle. On release of the weights it becomes the leading open-weights model, and Artificial Analysis notes the gap from the best open model to the best proprietary one narrows from 12 points to 3.

2. The headline number, and the other two

Terminal-Bench 4.0 is the benchmark the launch was read on, because it is agentic, it is hard, and K3.1 Pro appears to win it outright. Here is the problem.

Three figures, all published during launch week, all correct for the thing they measured:

The spread is 14.4 points. For comparison, the entire distance between first and fifth place on Moonshot's own Terminal-Bench table, from Claude Opus 5.5 at 66.4 down to DeepSeek V4.1 Flash at 42.3, is 24.1 points. One model's measurement uncertainty, across published sources in a single week, consumes well over half the range the leaderboard is trying to resolve.

Two of those three numbers come from Moonshot's own materials. The 12.9 point difference between them is the effort setting alone, max against high, on one harness. That is not a discrepancy to be caught out on; it is a documented property of a model with a reasoning dial. It does mean a reader who takes 65.7 and a reader who takes 52.8 are both quoting the vendor correctly and will reach different conclusions.

3. Where it stops being a rounding difference

The third number is the interesting one, because it changes the ordering.

On Artificial Analysis's leaderboard, which runs "all 66 Terminal-Bench 4.0 tasks with the mini-swe-agent harness" and reports "pass@1 averaged over three repeats per task," the top three are:

On Moonshot's own table, run inside Claude Code at max effort, Claude Opus 5.5 scores 66.4 per cent and K3.1 Pro scores 65.7.

So Claude Opus 5.5 is 59.6 in one harness and 66.4 in the other, a swing of 6.8 points with identical weights, and the ordering between the two models inverts. Under mini-swe-agent, K3.1 Pro leads Opus 5.5 by 7.6 points. Under Claude Code, Opus 5.5 leads K3.1 Pro by 0.7.

Neither party is doing anything improper. Both disclose their harness. The claim that K3.1 Pro achieves the highest Terminal-Bench 4.0 score of any model is true of the configuration Artificial Analysis ran, and the claim that Opus 5.5 is narrowly ahead is true of the configuration Moonshot ran. What does not survive is the sentence underneath both of them, the one a reader supplies for free: that one of these models is better at terminal work than the other.

4. Why the harness moves the number

This is not a Kimi-specific artefact, and it is not new. It is measured, and the literature on it is explicit.

The August 2026 survey of command-line agents states the position plainly: "The harness is part of the evaluated configuration." Its list of what a harness controls is the list of things a benchmark score silently depends on: "Observation formatting, approval, sandboxing, retry limits, context truncation, wrappers, and recovery affordances can alter performance under a fixed model." Its conclusion is the sentence that should sit at the top of every launch table: "Model comparisons therefore depend on the harness conditions under which behavior is produced."

The same survey measures it. On matched task identifiers, with the model held fixed, the gap between the best and worst agent system was 21.25 percentage points on Claw-SWE-Bench Lite against at most 7.00 points on SWE-bench Lite. More pointedly, the relative ordering of two harnesses, OpenHands and mini-SWE-agent, reverses between those two benchmarks. The ranking is not stable under a change of either axis.

A June 2026 position paper puts a number on the ceiling: "On this fixed task distribution, differences of 20 percentage points or more appear across harnesses," which it describes as a range comparable to differences between model generations. That is the finding that matters commercially. A harness choice can be worth a model generation.

We ran into the same wall reading the Astra and Fable launch tables in GPT-6 Astra against Claude Fable 5.1 on coding, where two points on Terminal-Bench sat inside the range where harness details and run to run variance matter. What is different here is that the variance is no longer something to allow for in the margin. It is 14.4 points wide, and it decides the ranking.

Artificial Analysis has tested this on its own stack and found the opposite in one case, reporting that it did not observe a significant performance difference between the Codex harness and its mini-swe-agent reference across effort levels, except at low effort. That is worth holding onto, because it means harness sensitivity is not uniform. It is a property of the model and harness pair, not a constant you can correct for. Which is precisely why it cannot be corrected for from the outside.

5. The part nobody will say out loud

Almost every coding score in Moonshot's launch table was produced inside Anthropic's Claude Code: Terminal-Bench, SWE-Marathon, DeepSWE v1.1 and ProgramBench are all confirmed, at max effort, temperature 1.0 and top-p 0.95.

Moonshot measured its own model inside a competitor's agent harness, and published the result as its headline.

The charitable reading is also the correct one: Claude Code is a strong, widely used harness, and running in it makes the comparison against Claude models more like-for-like rather than less. The uncomfortable corollary is that the configuration which produced the launch numbers is not the configuration most Kimi users will run. Someone calling kimi-k3.1-pro from their own scaffold, or through an API gateway, is not running the setup that produced 65.7 per cent, and has no published number for the setup they are actually running.

That is a gap worth naming, because it is the gap between a benchmark and a procurement decision.

6. Where the price went

The rate card fell by exactly a third. Input went from 3 dollars to 2, output from 15 to 10, cached input from 0.30 to 0.20. On any spreadsheet built from rate cards, K3.1 Pro is 33 per cent cheaper than K3.

Artificial Analysis measured what it actually costs to finish their index, and it came out at 1.98 dollars per task, roughly equal to K3. The model also emits more: 55K output tokens per Intelligence Index task against 48K for K3, about 14 per cent more.

That arithmetic does not close on its own, and it is worth being precise about why. A 33 per cent rate cut against a 14 per cent increase in output tokens should land at roughly 24 per cent cheaper per task, not flat. Something is consuming the difference, and the published figures do not say what. The candidate that fits an agentic index is input: more tool-calling turns, each one re-sending the accumulated context, so input tokens grow faster than output tokens and the input side of the bill absorbs the cut. We cannot demonstrate that from the published numbers, and we are not going to assert it. What we will assert is the conclusion that follows either way, and it is the same conclusion we reached about K3 in Kimi K3 vs Claude: what tokens actually cost:

A rate card is not a price when the model decides how many tokens the task takes. Cost per finished task is the only figure with a procurement meaning, and it moved by nothing while the rate card moved by a third.

7. What improved, measured rather than claimed

Setting the harness problem aside, the third-party deltas from K3 to K3.1 Pro are large and consistent in direction, which is the pattern of a real post-training gain rather than a benchmark-specific one. All of these are Artificial Analysis's figures, K3 first:

The Terminal-Bench figures here are the rounded ones from the launch article; the evaluation page carries the same run as 67.2 per cent.

The Terminal-Bench line, 13 to 67, is the one that should make a reader pause rather than cheer. A 54 point gain on one agentic benchmark from post-training alone, while Humanity's Last Exam moves 5 points and CritPt moves 2, is the signature of a model that has been trained hard on a capability the benchmark measures. Moonshot says as much, attributing the gains to reinforcement learning across "hundreds of downstream domains." Whether that generalises to agentic work the benchmark does not cover is the open question, and no number here answers it.

The hallucination movement is the most credible item on the list, because it is adversarial and third-party: the AA-Omniscience hallucination rate fell from 53 per cent to 44 while accuracy rose only 3 points, from 48 to 51. A model getting more willing to decline is a different achievement from a model getting more right, and it is the one that matters for anything customer-facing. Moonshot's own internal stress test reports a larger fall, false claims from 23.4 per cent to 8.3, but that test selects prompts designed to trip models up and is unreplicated.

8. Our position

Three things, stated as positions rather than findings, because the sources do not state them.

A score is a claim about a configuration, and should be written as one. "67.2 per cent on Terminal-Bench 4.0" is not a fact about Kimi K3.1 Pro. "67.2 per cent on Terminal-Bench 4.0, mini-swe-agent, pass@1 over three repeats" is. Every number in this paper is written that way above, and the discipline costs nothing.

For a buying decision, the vendor's harness is the wrong harness, including when the vendor is honest about it. The useful experiment is your tasks in your scaffold at the effort level you can afford, and the launch table's job is only to tell you which two or three models are worth running that experiment on. On that job, this table did fine: K3.1 Pro has clearly earned a place in the shortlist.

The open-weights release is the event, not the benchmark. A 3 point Intelligence Index gap to the frontier, in a model you can host, quantise, fine-tune and audit, changes the architecture of a deployment decision in a way that a 0.7 point Terminal-Bench gap does not. If the weights land on 4 November as promised, that is the thing to write about, and the number to watch is not a leaderboard position but whether anyone can reproduce 65.7 per cent outside Claude Code.

9. Limits of this reading

We have not run Kimi K3.1 Pro. Every figure here is Moonshot's or Artificial Analysis's, and where they disagree we have said so rather than picked.

The three Terminal-Bench numbers are from three sources at different levels of primacy. The 67.2 per cent and the methodology quotations are first-hand from Artificial Analysis's own evaluation page. The 65.7 per cent, the 52.8 per cent and the harness attributions reach us through launch-week coverage of Moonshot's table rather than from the table itself, which we could not retrieve directly. If Moonshot's own figures differ from those reported, the specific numbers move and the argument does not, because the argument rests on the Artificial Analysis and Claude Code results for Claude Opus 5.5, which are 59.6 and 66.4.

Sources also disagree on the Intelligence Index field. Artificial Analysis's launch article places K3.1 Pro fourth behind three models; at least one contemporaneous account lists only Claude Opus 5.5 ahead of it. The difference appears to be whether "max with fallback" variants are included. The 55 is consistent everywhere.

The launch date is reported as both 23 and 24 October 2026. We use 23 October, which is the date on the Artificial Analysis article and on the first independent hands-on account.

Nothing here is a system we built or operate. We have no relationship with Moonshot AI, Anthropic or Artificial Analysis.

Sources

1. Artificial Analysis, Kimi K3.1 Pro launch benchmarks, 23 October 2026. The Intelligence Index score of 55, the fourth place, the K3 to K3.1 deltas, cost per task and output tokens per task.
2. Artificial Analysis, Terminal-Bench 4.0 evaluation. The 67.2 per cent, the 63.6 and 59.6 for Claude Sonnet 5.5 and Claude Opus 5.5, and the methodology quoted above.
3. Simon Willison, Introducing Kimi K3.1 Pro, 23 October 2026. First-hand confirmation of the API model ID, the three reasoning effort levels, the Highspeed variant and the weights commitment.
4. Jake Handy, Model Drop: Kimi K3.1 Pro. Moonshot's full benchmark table with harness and effort footnotes, the 52.8 per cent cost chart figure, and the internal hallucination test.
5. Terminal Agents: A Survey of AI Agents in Command-Line Environments, 20 August 2026. "The harness is part of the evaluated configuration," the 21.25 point matched-system gap on Claw-SWE-Bench Lite, and the harness ordering reversal.
6. Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering, 16 June 2026. "On this fixed task distribution, differences of 20 percentage points or more appear across harnesses."
7. From Question Answering to Task Completion: A Survey on Agent System and Harness Design, 14 June 2026. The taxonomy of what a harness is responsible for.
8. DeltaSelect: Affordable A/B Testing for Coding Agents, 17 September 2026. On comparing agent configurations without paying for full evaluation runs, which is the practical form of the experiment recommended in section 8.