This is an argument rather than a reading of someone else's work. The evidence behind it is partly our own: fifty four papers in this library spent a year measuring frontier models against their published claims, and the pattern in those measurements is what this paper is really about. Where it cites outside figures on hiring, those figures are weak and labelled as such.
In the four weeks to this writing, a frontier model shipped, a competitor shipped a point release, two of them changed their prices, and one of those changes altered which model is cheaper for agentic coding by a factor of nearly three at long context.
No procurement cycle absorbs that. No annual vendor review survives it. And the thing most companies do instead, which is to read a launch post and a leaderboard, cannot answer the question they are actually asking.
So somebody has to own it. Not a committee, not a quarterly review, a named person whose job it is to know which model runs which workload here, and to have the evidence. We think that is a role, we think most companies will have it within two years, and the argument below is partly about why, and partly about which of the obvious candidate roles will not survive.
1. Why the published numbers cannot answer it
This is not a complaint about dishonest vendors. The numbers are mostly fine. They answer a different question.
Three things we have measured repeatedly in this library:
Margins are often inside noise. When we read the structure agnostic distillation paper in latent distillation needs no shared grid, the headline improvement was 0.27 points, and the authors used an appendix to say that seed variation at their operating point was 0.1 to 0.2, so their own result read as on par. That was unusually honest. The general case is that a reader cannot tell a real gap from a seed from the table alone.
The aggregate indices measure the axis that moved least. We set that out in AGI talk measures the wrong axis: general reasoning scores moved roughly five points across a full generation of models, while token efficiency, scope adherence and the ability to operate software moved by multiples. A buyer reading an intelligence index is reading the flattest line on the chart.
The comparison is not run under your conditions. In GPT-6 Astra: what the footnotes say the nineteen footnotes mattered more than the table: one competitor's scores came from a variant with fewer safeguards, another model got a tuned developer message, and a third was absent from three benchmarks because it declines those questions, which is a product decision rather than a capability gap.
And when we ran the comparison ourselves in GPT-6 Astra or Fable 5.1 for coding, the benchmark margins were inside noise and the decision turned on something no benchmark reports at all: a four times difference in cache read pricing, which on a coding agent that resends repository context every turn dominated the bill.
That is the case for the role, stated plainly. The information that decides the question is not in the published numbers, and acquiring it is a standing job.
2. What a model testing manager owns
Not a tool. A decision, and the evidence for it.
Concretely, five things:
A fixed set of real tasks. Ten to thirty units of work drawn from what the business actually does, with the ambiguity and the awkward legacy cases intact. They do not change between evaluations, because the point is comparability over time.
One harness, one bar. Every candidate runs the same way, at comparable effort settings. Maximum effort on one model against a default on another is not a comparison, and it is the most common way an internal evaluation produces a wrong answer.
Three measurements, not one. Accuracy on those tasks. Speed in wall clock time to a finished unit of work, not tokens per second. And cost per completed task including retries and the time a person spent fixing the output. The third is the one that changes decisions, because a model that is cheaper per token and needs more turns is not cheaper, an argument we worked through in small language models: the agent economics.
A routing decision, written down. Which model serves which workload, and why. Most organisations should be running more than one, and the reason to write it down is that the reasoning expires.
A re run trigger. Every model release, every price change. This is the part that makes it a role rather than a project, and it is why the last node in the flow above loops back to the work rather than terminating.
Where it sits: close to engineering, reporting into whoever owns the AI budget. It is not a research job. The person needs to be able to run an experiment honestly and read a bill.
3. The AI workflows manager
The second role is less about models and more about the business, and we think it is the larger of the two in headcount terms.
Engineering can answer whether a step can be automated. Nobody currently owns whether it should be, which steps keep a human gate, what the escalation path is, and who gets called when an agent does something expensive at three in the morning.
That is an operations job, and it is the natural owner of the approval lane discipline we set out in governed autonomy: which actions need a human, and how review is made cheap enough that people actually do it rather than clicking through. A review that takes four seconds is not a control.
It also inherits the failure modes. An agent with broad standing credentials is the highest severity issue most deployments have, which is the argument in the agent security paper, and somebody has to be accountable for the fact that it is true of your deployment specifically.
The distinguishing question, and the one in the flow above, is not technical. It is what a failure costs. That is a business judgement, which is why this is not a job for the engineering team to absorb quietly.
4. The role we would not create
You asked about a fine tuning role. We would not create that title, and the reasoning is the most useful thing in this paper, because it generalises.
Fine tuning names a technique. Techniques get absorbed. The tooling around adaptation has improved to the point where, as we described in Tinker, Inkling and the return of fine tuning, a training run is priced per million tokens like inference and a rollback is detaching an adapter. When a technique becomes that cheap and that reversible, it stops needing a dedicated person and starts being something an engineer does on a Tuesday.
What does not get absorbed is the asset the technique consumes. The valuable input in adaptation is not compute, it is a correctly labelled record of your own work: logged trajectories, the awkward tail of failures, and a held out evaluation set built by somebody who did not build the training set. Almost no company treats that as an asset, and nobody owns it.
So the durable role is custodian of that record, and in most organisations it is the same person as the model testing manager, because the evaluation set and the training corpus are the same artefact viewed from two directions. Name the role after the asset, not the technique.
5. How roles actually form, and the test
The pattern is older than AI, and it is not that a tool creates a job.
Photoshop shipped in 1990 and did not create the job of retoucher, which already existed in darkrooms. What it created was a new set of things that could go wrong and a new question about who was accountable for the result, and the profession reorganised around that.
Site reliability engineering was coined at Google in 2003, when Ben Treynor Sloss was given a production team of seven engineers and described the discipline, in the opening chapter of Google's own SRE book, as what happens when you ask a software engineer to design an operations function. The title names whether the service stays up. It outlived every tool in the original stack.
Data scientist was settled in a March 2008 email thread at Facebook, where the team chose it over research scientist, with Jeff Hammerbacher there and DJ Patil using it at LinkedIn in the same period. Patil later told the story in the Harvard Business Review piece that made the title famous, Data Scientist: The Sexiest Job of the 21st Century. The phrase data science was older, going back to Peter Naur in 1974, but the job dates from the moment somebody needed a name for accountability over what the data says.
DevOps got its name from Patrick Debois and the first DevOpsDays in Ghent in 2009, which grew out of an Agile Infrastructure session in Toronto the previous year that two people attended. Debois has since said he did not coin the term.
In each case the title names a decision somebody can be held to, not a tool and not a technique. That is the test, and it is the branch in the flow above.
6. The counterexample, which is the strongest evidence
Prompt engineer is the control group, and it failed.
Reporting through 2026 is consistent in direction and unreliable in magnitude: the standalone title has declined somewhere between 30 and 73 percent from its 2023 peak depending on which source you believe, while postings tagging prompt engineering as a skill rose sharply, one analysis putting it around 250 percent. Microsoft's own Work Trend Index is reported to have dropped the title from the roles employers planned to hire between its 2024 and 2025 editions, which is the closest thing to a primary source in that set and still not a published number we can quote.
We would not lean on any of those percentages. They use different baselines, they measure different things, and most of the sources publishing them sell courses. What survives the scepticism is the shape: the title went down while the skill went up, which is precisely what absorption looks like.
And it failed the test. Prompt engineering named a technique. Nobody could be fired for failing at it, because nobody could say what success was. Compare the proposed role here: a model testing manager can be fired for the company running an expensive model on a workload a cheaper one handles, and for not knowing it. That is an accountability.
Apply the same test to any title your industry is currently inventing. If it names a vendor, a tool or a technique, it has perhaps two cycles.
7. The regulatory floor, which is lower than the argument
Worth being accurate rather than using compliance as a sales lever.
In the EU, Article 4 of the AI Act has required providers and deployers to take AI literacy measures since 2 February 2025, and Article 26 requires deployers of high risk systems to assign human oversight to people with the necessary competence, training and authority, with the main Annex III obligations applying from 2 December 2027. Those duties presume a named person. We set out the full position in the EU AI Act 2026 guide and what enforcement actually looks like in the first month of enforcement.
But most business AI is not high risk, and we would not argue these roles on compliance grounds. The regulation makes a floor visible. The money is the reason.
8. What to do about it this quarter
You probably cannot hire two people. You can create the accountability, which is the part that matters.
- Name somebody. One person, named, who owns which model runs where. It can be twenty percent of an existing engineer's time. What cannot be part time is the accountability.
- Build the task set before the first comparison. Ten real units of work. This is the asset; everything else is method.
- Run the current state through it. Most organisations discover they are paying frontier prices for classification and extraction, which is the single most common finding and the fastest payback available.
- Write the routing decision down, with the date and the numbers behind it.
- Put a re run on the calendar against model releases rather than quarters, because the releases do not wait for quarters.
The companies that do this will not look more advanced than the ones that do not. They will look cheaper and less surprised, which is the same thing eighteen months later.
If you want the task set and the first comparison built properly rather than argued about, talk to us, or see how we build systems for businesses.
9. Sources and limits
- Beyer, Jones, Petoff and Murphy, editors, Site Reliability Engineering, Google. The introduction carries Ben Treynor Sloss's definition and the 2003 origin, and Part II on principles is the clearest statement of a role defined by an accountability rather than a toolset.
- Davenport and Patil, Data Scientist: The Sexiest Job of the 21st Century, Harvard Business Review, for the naming of the role.
- Devopsdays Ghent 2009, the event record itself, for the date and the organisers.
- Article 4, AI literacy and Article 26, obligations of deployers of high risk AI systems, on the European Commission's AI Act Service Desk.
- Microsoft, Work Trend Index, for the annual role and skill reporting discussed in section 6.
The historical claims are reasonably well attested and still worth checking rather than trusting. The Facebook email thread is secondhand in every account we could find, the phrase data science goes back further than the job, to Peter Naur in 1974, and Debois has himself said he did not coin the word DevOps. Dates in the blog literature conflict by a year in several places.
The hiring figures are weak and we have said so in the text. Percentage declines for the prompt engineer title range from 30 to 73 percent across sources that use different baselines and different measures, and most of them sell training. We use them only for direction.
The core claim of this paper is a prediction, not a measurement. We believe most companies will create these accountabilities because the cost of not having them is already visible in our own client work, but nobody can show you a dataset proving a role will exist. What is measured here is the premise underneath it: that published benchmarks do not answer the procurement question, which fifty four papers in this library were written establishing.
This note sits in our Operations research, alongside the governance thread. The full library is at Research.