A report landed with one big number in it: 54% accuracy over 500 companies. Below it, the conclusion that follows from a number like that: the algorithm needs a lot of work.
The system under test was an industry classification service: it takes a company, searches for its web presence, identifies it, reads what it actually does, and maps that activity to an industry code validated against the taxonomy’s official hierarchy. It takes on the order of a minute per entity. The test was run externally, against the API, with no involvement from me.
Let me be honest about my first reaction, because it’s the part I least enjoy telling: I didn’t doubt the report. A percentage with decimals over a sample of 500 carries an authority you don’t question on first read. I spent a good while thinking about what we’d done wrong.
The first clue
When I re-ran the set on my side and sat down to look at the disagreements one by one, a pattern showed up that didn’t fit: the gold set labels could be guessed without looking at the company at all. The name was enough. If the registered name contained a word like logistics, the label was transport. If it contained solar, energy. If the name was opaque — an acronym, a surname, an invented brand — the label fell into some generic services category.
That correlation is the whole finding. A label you can predict from the name carries no information about the company: it carries information about the name. And plenty of companies don’t do what their name suggests. A consultancy called something Mining Services that actually sells software to mining firms gets classified as mining if all you read is the sign, and as software if you open its website. The cases where the name misleads are precisely the hard ones — which is to say, the only ones worth measuring.
I asked. The answer was direct and undramatic: the gold set had been generated by pasting the list into ChatGPT.
Two different processes, one score
That’s the heart of it, and it isn’t a matter of wounded pride. The system under test spends a round per entity on official-registry enrichment, web search and reading the real activity before it assigns anything. The grader read a column of names.
This wasn’t system versus truth. It was a process with research in it against a process without, and the one handing out the grade was the one with less information of the two. A grader with less access to the world than the system it evaluates doesn’t measure the system: it measures the distance between two methods, and calls everything the system knew beyond it an error.
There’s a line we ended up repeating among colleagues every time this came up, and it captures the problem better than any analysis of mine: if ChatGPT were good enough to build the gold set, this tool wouldn’t be needed and we wouldn’t have built it. The reference set assumed away exactly the problem it was meant to measure.
The nuance: this isn’t a model problem
This is where the article could turn into “general-purpose LLMs can’t label data”, and that would be false.
A general-purpose model, one company at a time, with search enabled and time to browse, is a perfectly respectable baseline. I know because it was the project’s starting point: the founding question of the project was literally how far does this get done by hand in a chat, and what do we have to build to beat it consistently, auditably and at scale?. The answer to the first half was not bad at all.
What breaks isn’t the model, it’s the mode. Five hundred rows in a single request spread a tiny reasoning budget across each row and, more importantly, leave no room to go look anything up. The model then does the only thing it can do with what’s in front of it: infer from the available text, which is the name. That’s not a hallucination or a capability failure; it’s the correct answer to the question it was actually asked, which wasn’t the question anyone thought they were asking.
And this boundary moves. An agent with a browser and a per-case budget already does a real share of that research today, and will do more a year from now. Which is why the useful criterion isn’t was it done by an AI? but was there research per case, and how deep?. That second question will still hold after models have turned over three times.
The rule I’m taking away
There’s a fairly clean line here between two uses we tend to lump together: AI as a judge and AI as a researcher.
A model evaluating another model works well when the judgment closes with what’s already on screen: is this answer consistent with the context? does it respect the format? which of these two is better? does it follow the instructions it was given? There an external grader even brings something we don’t have: a perspective with no affection for the system. That’s a legitimate and reasonably well-studied use: it’s the ground where the idea of models as judges took hold, comparing answers and measuring agreement with human preferences.
It stops working when the correct answer isn’t on screen but out in the world: in a companies register, on the firm’s website, in a news item from three months ago. Then the grader has to do at least the same work as the system it’s grading. Not similar work: the same or more. If it doesn’t, the metric it produces measures the grader, not the system. That isn’t incidental: surveys of the LLM-as-a-judge literature put knowledge problems among its central limits — stale information, domain gaps, and confidently invented claims.
The one-line test I apply now before accepting any evaluation:
Did the grader have access to more information than the system it graded, or less?
With more, the evaluation can be harsh and still be useful. With the same, it’s arguable but informative. With less, it isn’t an evaluation.
The boring errors that were also there
The gold set was the big problem, but not the only one. The comparison wasn’t built well either:
- Order-sensitive. An entity can receive several codes, and an answer was counted as wrong if they came back in a different order than expected.
- Blind to compatibility. These taxonomies contain codes that are legitimately compatible: a company can fall under two and both be right. One was counted as a hit and the other as an error.
- Binary. No distance in the hierarchy. Missing the branch entirely and missing a subcategory within the right branch scored exactly the same.
Worth noting about all three: they push in the same direction as the bad gold set. A reference set inferred from names plus a strict comparator can only subtract points, never add them. When every bias in a measurement points down, the resulting number isn’t merely “noisy” — it’s biased, and biased in the expensive direction.
What an honest number cost
Redoing it properly was manual work, and I found no shortcut. Case by case, starting with the disagreements. Leaning on a good model, one at a time and slowly, helps a lot — considerably more than in batch, which is precisely the point — but even then it doesn’t come out clean: there were entries I had to open by hand because the automated call didn’t sit right with me, and some cases remain defensible under two different codes.
The result was 75% on a conservative reading, and up to 85% if you resolve the grey zones favourably. Around twenty points above the report.
And the part that also needs saying: the system did not come out of that review looking perfect. There were real failures, and those were the useful material in the whole episode. What didn’t exist was the diagnosis. A 54% says this is broken, rethink the approach. A 75% with a list of concrete cases says this works, and here are the ten things to sharpen. Those are two completely different product decisions, and one of them would have been very expensive.
That 75% shouldn’t be read as a fixed property of the system either. This batch was a particularly hard one, and achievable accuracy depends heavily on composition: what share of the companies have a web presence of their own, of what kind, and in how much detail. A set of businesses with live sites describing what they do classifies far better than one loaded with holding companies that leave almost no public trace. Comparing the percentage of two different batches as if they measured the same thing is another version of the same underlying error.
And out of the review, together with the client’s feedback, came the conclusion that most changed how I read the metric: a wrong result the system itself flags as low confidence is not the same kind of failure as a wrong result held with high confidence. The first gets routed to human review and the workflow does exactly what it should; the second is the one that actually causes damage. Flat accuracy counts them the same, and that’s probably the most serious limitation of measuring this with a single percentage.
Why labelling got expensive
This incident is a domestic version of something happening at industrial scale. In 2026, Meta reassigned thousands of engineers — on the order of 6,500 people in its agent data organisation, according to reports — to producing and labelling training data, with a considerable internal revolt and a partial reversal afterwards. You can argue about the management of it; the underlying signal is harder to argue with: someone decided that paying frontier engineering salaries to label data was worth it.
That reading matches this episode. Running an evaluation is cheap. Building the truth you compare against is not. It’s the part that remains slow, human and boring, and that’s why it’s being paid for better and better. Pasting the dataset into a chat in batch isn’t a fast version of that work: it’s the operation of skipping it entirely while keeping the appearance of the result. And that appearance is worse than having nothing, because a false number does make decisions — nobody rethinks a roadmap over an empty folder.
Before you believe a metric
The five questions I now put to any evaluation before arguing about its result:
- Who built the gold set, and with what access to information? If the answer is “a model, in batch”, you already know what it measures.
- Can I predict the labels with a dumb heuristic? Take the shallowest attribute of each case — the name, the first word — and try to reproduce the gold set. If you can, the gold set is that heuristic.
- What exactly counts as correct? Order, multiplicity, synonyms, distance in the hierarchy. Many quality arguments are really comparator arguments.
- Were disagreements reviewed by hand, or only counted? An unreviewed disagreement isn’t an error: it’s a disagreement.
- Among the reviewed disagreements, who wins? If the system wins a high share of them, the problem isn’t the system.
Through this whole episode, the system was under suspicion from minute one and the gold set was never under suspicion for a second. That asymmetry is the interesting part: we treat whatever serves as the reference as audited precisely because it serves as the reference. When an evaluation tells you your system is bad, the cheap first hypothesis isn’t to rebuild the system. It’s to open the gold set and check whether anyone went and looked.