Nineteen of twenty patched the symptom

In the previous article I described a habit I kept running into: when a system has a language model inside it and something goes wrong, the explanation drifts towards the model. The sampling. The nondeterminism. Something nobody wrote and therefore nobody has to fix.

That was an observation from work, which is a polite way of saying it was an anecdote. So I built a way to measure it, and the measuring turned out to be harder and more interesting than the result.

The setup

I built a small classifier that reads a repository’s documentation and assigns it a category from a hierarchy, with a confidence and a justification. Real repositories — fifty of them, low-star and recent so that no model has them memorised. It runs on a real model. It has a search budget, a context window that truncates, a retrieval step, and a trace.

Then I planted a fault in it, ran the same batch five times, and got a table where a handful of projects change category between passes even though nothing changed between passes.

That table is what the agent sees. The only thing I varied is whether it also gets the code.

Scenarios were frozen with a hash before any of this ran, the people coding the answers never knew which group a response came from, and the statistics were computed from the raw JSON by a script that ships with the repo. I’ll come back to why all that ceremony mattered.

The first fault: the order of retrieval

The planted fault: the search that retrieves documentation resolves score ties using an identifier derived from the request, so each pass hands the model a different set of documents. The variability comes entirely from the input. The model’s sampling has nothing to do with it.

Twenty agents saw the table without the code, twenty with it.

The headline isn’t the attribution. Blaming the model outright happened four times out of twenty, and at p = 0,053 that doesn’t clear the usual bar. The strong formulation of my own thesis — the agent blames the model — did not survive contact with the data, and I’d rather say so plainly than round it into significance.

What did survive is the behaviour. Nineteen of twenty, with no way to see the system, set about damping its output — voting across retries, pinning the temperature, averaging the instability away. With the code in front of them, nine. You don’t have to say the model is at fault to treat it as if it were: it’s enough to stop looking for a cause and start smoothing the symptom.

The second fault: where blaming the model is half right

The first scenario has a weakness I could see from the start. Nothing in it makes sampling a reasonable explanation — it’s just the lazy one. A fair test needs a case where a competent engineer could reach that conclusion in good faith and still be wrong.

So I built a second one, and this is the part I’d defend hardest.

which project goes first depends on the batch’s iteration order

Why this one is fair: the symptom is “it fails in the batch, and reproduces fine on its own”, which is the canonical signature of nondeterminism. The affected projects wander from pass to pass. The justifications read like textbook hallucination — fluent, confident, describing features the project doesn’t have.

And here’s the part that makes it honest. Of the fifteen label changes in the table, fourteen are contamination and one is genuine model sampling — a project that was never served anyone else’s documents and moves anyway, confirmed by resampling its prompt twenty times. For that one, “it’s the model” is the correct answer.

So the rubric has two opposite fields that can both be true of the same response: blaming sampling for the fourteen is the error, and attributing the one correctly is the win.

Making the wrong explanation reasonable is what moved the number: from 4/20 at p = 0,053 to 6/20 at p = 0,0101. The reflex isn’t summoned by laziness. It’s summoned by a situation where it half fits.

One result went the opposite way from what I expected: proposing to fix the temperature was more common with the code than without (8/20 vs 2/20). Reading the responses explains it — with the source in hand they can see the residual sampling is real, and they propose pinning it as part of a fix. It’s an informed suggestion, not a reflex, and it’s a good reminder that a single checkbox measures behaviour badly.

Nobody asked. A hundred and forty times.

Across every scenario and every group, not one response asked for the information it was missing before reaching a conclusion.

Not once. Every package deliberately shipped shallow logs — input and final answer only — while the full trace existed and would have been handed over on request. The brief said so: you can ask for whatever you’re missing. All of them diagnosed first and listed what they’d lacked at the end, after the conclusion was already written.

The obvious explanation is social: being shown a table of outputs frames the job as analyse this, and asking for more input reads as refusing the job. That’s testable with one sentence, so I tested it. Sixty more responses, same frozen scenario, with the brief rewritten to say that asking is a complete answer, preferable to a hypothesis you can’t check — and closing with asking is not leaving the job half done.

A hundred and forty out of a hundred and forty. Telling them in as many words that asking is not shirking changes nothing about whether they ask.

What it does change is the shape of the answer. Hedging doubles. Given explicit permission to ask, they don’t ask — they cover themselves, flagging the conclusion as provisional instead of doing the one thing that would make it firm.

And the other half of the explanation is what they do instead. They build their own measurement: a script, a sweep over the corpus, a synthetic reproduction of the pipeline. On the stronger tier, 18 of 20 do this, against 7 of 20 on the weaker one (p = 0,0004) — the largest capability difference in the whole study, and it isn’t about willingness to ask, which is zero everywhere. It’s about being able to manufacture the answer without asking.

So it isn’t incuriosity about data. Several of these responses did genuinely rigorous work to get data. Asking for it simply isn’t in the repertoire, and whoever can, fabricates it instead.

patched the symptom 19 9

found the cause 3 20

asked for the data 0 0

A finding that didn’t survive

I’ll spend a paragraph on something that isn’t in this article any more, because how it left is the point.

Half of each group ran on a stronger model tier and half on a weaker one. In the first round, with ten responses per cell, the stronger tier appeared to find the cause less than the weaker one — 4/10 against 9/10, p = 0,029 — and I had a tidy explanation ready. The stronger responses had audited the corpus document by document, verified that no snapshot contains another project’s material (which is true, because the contamination happens at runtime, not in the data), and concluded from that a cache couldn’t be responsible. Correct reasoning, correct check, false conclusion.

It was a good story. So I doubled the sample to twenty per cell, and it evaporated: 2/20 against 5/20, p = 0,20. It was noise, and a p of 0,029 with ten per cell is exactly the kind of number that looks like a finding and isn’t.

The one tier difference that does hold is the one above — the stronger tier builds its own measurement far more often. That one I’d defend.

What doesn’t hold

• With the code, 20/20 found the first fault. That group doesn’t discriminate, and I knew it wouldn’t before running it: reaching a middling difficulty there would have required fabricating metadata — assigning repositories to package registries that don’t list them, staggering capture dates — and I ruled that out. A study about agents cutting corners can’t cut that one. The with-code group is the control, not the measurement.
• The literal form of the thesis remains unmeasured. Blames the model for what it wouldn’t blame itself for requires the system to be the agent’s own. In all four groups it audits someone else’s code. What I measured is an asymmetry of material, not of authorship.
• One certification came in at 18/20 rather than the 19/20 I’d set as the bar, and the fix removes all fourteen contaminations but leaves three residual changes — the 0,7% sampling floor that no cache key touches.
• The rubric hardened between rounds, and the same condition scores 13/20 under the old definition and 3/20 under the new one (p = 0,0015). The second demands that a response describe the mechanism, and counts listing the cache among several uncommitted hypotheses as a miss. Numbers from different rounds of that field cannot be pooled or compared, and none in this article are.

The part that took longest

Three attempts failed before one worked, and the failures are more instructive than the result.

The first gave agents the code with a planted bug and asked what went wrong. All eight found it. A scenario nobody fails measures nothing.

The second showed a batch of bad classifications with no variability at all — and the thesis is about variability, so there was nothing for anyone to attribute to sampling. I’d built a case that couldn’t contain the phenomenon.

The third — a rewrite that tried to make the fault hard by using concurrency — was killed in review before it ran, on a point I hadn’t seen: if the entropy comes from network jitter against the inference service, then “the variability comes from the model layer” is true, and the rubric would have scored a correct answer as an error. Hard and deterministic pull in opposite directions, and you have to resolve that on purpose.

What survives all three is a design rule I’d now apply to anything of this shape: the fault has to be something the model provably cannot have caused. In the final scenario that’s guaranteed by one fact — the same project, asking for the same sections, receives different prompt bytes across passes. Sampling cannot change the bytes of a prompt. It isn’t a judgement call by whoever grades the answers; it’s arithmetic.

And the ceremony earns its keep. Scenarios hashed before the run so they can’t be tuned after seeing results. Coders blind to which group a response came from. Calibration data kept separate from confirmation data rather than merged to inflate the sample — which would have been free, and would have been exactly the shortcut this experiment exists to detect in others.

---

Code, data, and the script that recomputes every number in this article: blaming-the-model. The observation this started from is in the previous article.