Quality is measured against cases, not opinions

Most organisations deployed faster than they built any way to judge what they deployed. A pilot can absorb that, because a handful of people read the output and form a view between them. The second deployment cannot. The population is past the point where anyone is reading it, and the first question finance asks is whether the thing works.

The three answers, and why each one fails

“Users like it.” That is adoption. It tells you people opened the tool and came back. It does not tell you the answers were right, and the two come apart more often than anyone expects. Plenty of reports were popular and wrong.

“Accuracy is 94 percent.” On which cases, measured when, and against whose judgement of correct? A figure without those three things attached is not a result. It is usually a number somebody produced once, on a sample chosen by whoever built the system, and it has been repeated in slides since.

“We have had no complaints.” Silence is the weakest of the three. Someone who gets a bad answer rarely files a ticket. They quietly stop using the tool and go back to doing the work by hand, which looks like nothing at all on a dashboard and reads as acceptance. Entry 007 covers what that refusal usually means and how to read it.

What the record shows

Direct evidence on how enterprises evaluate their own deployed systems is thin, and it is better to say that than to reach for a figure that sounds like it.

What can be checked is the measurement culture upstream. Stanford's 2026 AI Index reports that almost all leading model developers publish results against capability benchmarks, while reporting against responsible-AI benchmarks stays sparse. Read that as context rather than proof of anything about your own programme. If the organisations whose entire product is the model publish thinly on everything except capability, the enterprise deploying it has no example to copy.

The other series you can check yourself is the AI Incident Database, maintained by the Responsible AI Collaborative, which recorded 362 documented incidents in 2025 against 233 in 2024.

Handle that pair carefully, because the database is populated by public submission. It counts what somebody took the trouble to write up, which is not the same as what happened, and a rise of that size reflects attention as much as incidence. The denominator is unknown and the database does not claim otherwise. Taken for what it is, it still says that the field's own record of failure depends on people choosing to keep one.

Why the existing testing muscle does not transfer

Enterprises have a decade of practice testing deterministic software. The same input produces the same output, a test passes or it fails, and a regression is unambiguous. Almost none of that carries across to a system whose output is a judgement.

Two properties break it. The same prompt can produce two different acceptable answers, so exact-match assertions are useless. And correctness is often a matter of degree rather than a boolean, so the interesting question is not whether an answer is wrong but whether it is worse than the one a competent person would have given.

Teams that notice this usually respond by testing less, because the tooling they know does not fit. The answer is a different instrument, not fewer of them.

What to build

A set of real cases with agreed answers, drawn from the people who do the work today.

A few hundred is enough. Teams reach for a few thousand and stall, because the cost is in the agreement rather than the collection. Sample the actual distribution of work, including the awkward tail, and resist the urge to fill it with clean examples that make the score look better.

Run every change against it. A model version, a prompt edit, a retrieval change, a new vendor. The point is to be able to say by the end of an afternoon whether a change made things better or worse, which is a capability most programmes do not currently have at any timescale.

Then argue about the cases where the expert and the system disagree. Those arguments are the valuable part. They are where a function discovers that two of its own people would have answered differently, and that nobody had ever written down which one was right.

The set is worth more than the score

Building the cases forces a function to state what good means. Most have never done it. The knowledge sat in individual heads, was applied consistently enough to pass for a standard, and turns out on inspection to be contested at the edges.

That written record keeps its value through everything downstream. Three programmes this year changed their model, their vendor, or their retrieval layer, and in each one the thing that made the change survivable was a set somebody had built early and nobody had wanted to fund. The models underneath are the replaceable part. The cases you agreed to measure against are what makes a vendor change a decision you can take on evidence, rather than a leap.

A programme that cannot tell whether last week's change helped is not shipping carefully. It is shipping on impressions, and it will keep doing so until somebody writes the cases down.