When the Fact Stops Being True

A replication of SKILL.state: Scalable Long-Horizon Agent Skills (Badhe, Tiwari and Chung, accepted at EMNLP) across two models and 500+ episodes. Every headline number here is a count of decisions, not an average of episodes.

An agent reads an event at step 10: the pallet you filed at step 3 was never actually put away; that shelf is empty. Twenty steps later it has to decide where to store the next pallet. The correct answer is the shelf the correction freed.

With the entire transcript in its context — the original record, the correction, everything in between — Claude Haiku 4.5 gets that decision right 3 times out of 44. Given a 200-character JSON state object instead, and no transcript at all, the same model gets it right 44 out of 44.

That is the strongest effect in this replication. It is also the paper’s own claim: its Experiment 3 reports that history-based runtimes hallucinate for five to eight turns after a correction while explicit state recovers in zero. What is new below is the unit of measurement, not the sign.

What the paper proposes

SKILL.state replaces the append-only conversation history of a ReAct-style agent with an explicit, mutable execution state. At each step the model receives the procedure P, the current state Σ_t, and the latest observation O_t. It replies with a JSON patch and an action. The patch is validated and merged, Σ_{t+1} = Σ_t ⊕ ΔΣ_t, and the reasoning that produced it is discarded. Nothing accumulates.

The claim has two halves: better accuracy on long procedures, and a prompt that stays O(1) instead of growing O(T).

The reasoning that produced the patch is discarded. In SKILL.state, everything the agent will ever know about step 3 has to be inside the state object by step 4 — or it is gone.

SkillExecBench has no public code, so the environment here is a reimplementation from the paper’s §4.1 description — a 500-shelf warehouse with dense pipe-separated event records, matched on context density rather than literal content, and running 1.2–1.5x heavier than theirs by average prompt size.

One difference is deliberate and worth stating up front: their Table 1 runs on Gemini-3-Flash, with Gemma-4-31B-it and Qwen-3-8B-it elsewhere in the paper. This replication runs Claude Haiku 4.5 and Claude Sonnet 5. Where a result here disagrees with theirs, the first candidate explanation is the model family, not the method — and saying which of the two it is turns out to be most of the work.

Half of it replicates

The cost curve reproduces exactly. Average prompt at T=50, in the same unit the paper reports — characters: SKILL.state 2,157, flat from T=10 (2,136) to T=50, against their 1,773. ReAct: 16,437 and growing linearly, against their 11,931. O(1) against O(T), as advertised, at 1.2–1.4x their density.

The accuracy half does not, and the interesting part is that it does not at any horizon they tested. Their degradation is a scaling effect — ReAct falls from 0.90 at T=10 to 0.74 at T=200 — so the only honest way to check it is to run their whole range.

At T=200 the transcript arm is holding a 48,000-character prompt and 690 actionable events, and it misses one decision out of roughly 600. On Gemini-3-Flash the same arm misses a quarter of them. Re-measuring the T=50 SKILL.state cell with 3 seeds × 6 repetitions gives 18/18 at exactly 1.000, zero deviation, so this is not a lucky run either.

The Memory row looked at first like an artefact rather than a result: its runtime is the only one that makes a second call per step, and at T=100 it lost 23, 14 and 2 replies out of 100 to the output cap on the three seeds, scoring 0.58, 0.67 and 0.91 in that order. So it was re-measured with double the output budget:

Doubling the budget raised the truncation count — 23 replies cut off became 34 on the same seed — and left the score where it was. The model fills whatever budget it is given, and the low score is not what the cut-off replies were costing. Memory really does degrade, and it degrades further than in the paper.

The reason the other three hold is worth naming, because it governs everything after: in this task the load-bearing information is never far away. The freed shelf an agent has to reuse sits 1.9 positions from the top of the stack on average, at most 7. Making the horizon longer adds steps without moving information further from its use. If you want to measure whether a runtime remembers, T is the wrong knob — which is what the rest of this article is about.

The token count is not the bill

The paper compares tokens. Anyone running this compares money, and the moment prompt caching is on those are different quantities. An append-only transcript is the ideal cacheable prefix: every step re-sends exactly what it sent before, plus a suffix. A block that mutates invalidates the cache from the point it mutates.

There is one number you have to measure before any of this means anything: the minimum cacheable prefix. Below it, nothing caches at all. On Claude Haiku 4.5 it is 4,096 tokens exactly — a 3,984-token system block, sent twice, reads nothing from cache; a 4,116-token one reads all of it. So the answer depends on how long your procedure is, and both cases are worth having.

Short procedure — 1,491 tokens. Below the threshold, so no arm’s procedure caches. Only ReAct caches, and only because its accumulated transcript pushes the prefix past 4,096 on its own. SKILL.state’s advantage over ReAct falls from 7.54x in raw tokens to 1.39x in money. The orderings disagree too: by tokens it is SKILL.state < Memory < ReAct < Stateful; by money, SKILL.state < ReAct < Memory < Stateful.

Realistic procedure — 5,243 tokens. A field reference for the 112 fields the events actually carry, six exception rules, five worked examples. Real operating procedures look like this. Now every arm’s static half caches, and two things happen:

• SKILL.state gets 41% cheaper: $109 →$64 per thousand episodes. The procedure got three and a half times longer and the bill went down, because it crossed the threshold. Memory goes from saving 0% to 48%, Stateful from 0% to 22%.
• The advantage in money widens to 2.51x while the advantage in raw tokens narrows to 3.43x. In both conditions the raw token count is the wrong number to quote: it says 7.54x or 3.43x where the invoice says 1.39x or 2.51x.

The row to actually act on is Stateful. It sends almost exactly what ReAct sends. It puts a mutating state block in front of the transcript instead of behind it — which is where the paper’s own Appendix A.3 template puts it — and is billed $849 against$162 per thousand episodes. Same content, same task, same score of 1.00. A 5.2x difference, and it holds in both conditions.

On Sonnet 5, at 3x the input price, those become $2,546 against$486. Per thousand episodes, prompt order is a four-figure line item.

Where explicit state actually wins

The paper’s Experiment 3 tests what happens when the world changes underneath the agent, and reports it as a yes/no over four hand-built scenarios: history-based runtimes recover after five to eight turns, SKILL.state after zero. It is the one result of theirs that reproduces here, and the probe below measures it per decision instead of per scenario: the agent files a pallet at step t; at t+10 a correction says that put-away never completed and the shelf is empty. From then on that shelf is the lowest free one, and every subsequent decision depends on having applied the correction.

Two design choices make this measurable at all. First, the unit of counting is not the episode or the seed but the dependent step: each step after the notice whose correct action changes because of it. Second, the seeds are chosen by measured range before spending anything — a perfect-but-deaf agent, one that executes everything correctly and simply never applies the correction, defines the floor, and seeds differ enormously in how much room there is above it:

Seeds 4, 10 and 6 carry 22 dependent steps per repetition. Seeds 0, 1 and 2 carry five between them, and one of them carries none at all. Computing that table costs zero API calls.

Three things only visible when you count decisions rather than average episodes:

The transcript arm fails all-or-nothing per scenario. In Haiku, errors of any other kind are exactly zero across all six episodes: its only mistakes are the correction steps, and it misses them in blocks — 11 of 11, 7 of 7, 4 of 4. This is not an agent drifting. It is an agent executing a 50-step procedure flawlessly while never updating one fact. In Sonnet the same pattern appears bimodally: one episode applies all 11, the next repetition on the same scenario misses all 11.

More capability helps and does not solve it. Sonnet with the full transcript goes from 6.8% to 39.5%. It reconciles the contradiction far more often, and still loses three decisions in five.

Explicit state buys it, and in Sonnet charges for it elsewhere. Sonnet emitted 66 out-of-schema patches across 8 episodes, exhausting the retry budget on 10 steps that then produced no action at all, which cost it 21 errors of other kinds. Haiku emitted zero. Accuracy on the correction is 100% in both; the runtime’s reliability is model-dependent, and that is a property of the method rather than of the task.

The mechanism is unglamorous. Explicit state has exactly one place where the fact lives, and correcting it is the operation the runtime already performs every step. The transcript erases nothing: it holds the original record and its retraction simultaneously, and every subsequent step has to resolve the contradiction again from scratch.

Having a single place where the truth lives is an advantage precisely when the truth changes. That is the paper’s claim, not a counter-result. What this adds is a number under it — every dependent decision rather than four scenarios — and a replication in two models it never tested.

Where explicit state does nothing

The complementary probe: at step t the environment announces a shelf is quarantined. At t + k that shelf is the lowest free one and the correct action is to skip it. The dependent step and the shelf are identical for every k; only the distance between the information and its use moves. Nothing gets contradicted — the fact simply has to survive.

Here explicit state, by itself, does nothing at all. At k=40, Haiku with a state object and no field for the notice: 0 out of 24. What changes the outcome is not the runtime but where the fact is allowed to live:

Three interventions, measured paired on the same seeds:

• A schema field that names the fact (quarantined_shelves) takes Haiku from 0/24 to 100% and Sonnet from 12% to 75%. It works because the schema’s designer anticipated exactly this fact — which is one of the three settings the paper’s Limitations section declares and does not measure (a state update that depends on an observation whose relevance was not recognised when it was read), now with a number attached.
• A generic free-text field (notes, no indication of what to put in it) scores 5/24 = 21% on Sonnet, confidence interval 9–40%, statistically indistinguishable from having no field at all. On one seed it scores 0/8, worse than nothing.
• Re-injecting the standing fact into every observation takes Haiku from 0/24 to 24/24 and Sonnet from 12% to 83%, and requires anticipating nothing.

And the last one splits further. The re-injected banner carries three of the original notice’s fifteen fields, hoisted to the top. Repeating the entire original notice verbatim instead, same position, same header, 981 characters instead of 67, scores 16/24 = 67%. All three Wilson intervals are disjoint, and the distilled version wins on all three seeds.

So: making the fact available recovers two thirds of the failure. The remaining third is distillation — with the same fact present at every single step, buried among fourteen fields of operational metadata, the agent overlooks it one time in three.

The practical version, scoped to what was measured (one environment, k=40, the three-level comparison on Haiku only): if a fact stays true across many steps, re-inject the field, not the record. A system that replays the whole document into context leaves a third of the failures on the table. This is the same shape as the finding in The Scaffolding You Pay For — the intervention that survives is the one that changes what the model is looking at, not the one that adds structure around it.

Four checks this kind of experiment needs

An experiment on agents produces numbers whether or not it is measuring anything, and when it goes wrong the output is not noise — it is a clean result. Two findings here were fully written up, tables and non-overlapping intervals included, before these four removed them.

• Compute the floor before the effect. Simulate an agent that is perfect except that it ignores the thing you are measuring. On seeds 0, 1 and 2 that agent scores 0.931, 0.893 and 1.000, so the largest possible effect there averages 0.06 and one seed carries no information at all. A separation of +0.199 measured on those seeds was arithmetically impossible before anyone asked what caused it.
• Count steps with no action separately from steps with a wrong action. That +0.199 was truncation: the transcript arm lost 8 to 19 replies out of 50 to the output cap, and a reply cut off before its Action: line scores as an error. Raising the cap does not fix it — 600 tokens gave 19 truncated replies, 1,500 gave 11, 4,000 gave 18. A fixed output budget penalises the arm whose prompt grows, and the paper reports ReAct degrading with T without reporting truncation.
• Decompose the score by the type of step it averages. A second environment reported a sign-flipping interaction between models, with non-overlapping intervals on both separations. Instrumented, all four cells got its load-bearing rule right; only 5 of 34 actionable steps tested that rule, a blind policy scored 0.853, and the whole reported effect lived in routine steps. The section was withdrawn.
• Know your noise floor. Same seed, byte-identical prompt, eight repetitions: one seed alternated hit and miss eight times running. Single-step accuracy carries tens of points of sampling noise; an average over ~170 events carries almost none; token accounting carries none.

What I would take from this

• The advertised axis is the wrong axis. How much context you keep barely moves accuracy on a task where the information is nearby. What moves it is whether the load-bearing fact is present, current, and distilled at the moment of the decision.
• Explicit state earns its place when facts get invalidated. One location to correct beats a transcript holding a claim and its retraction — 93/93 against 18/82, in both models. If your agent’s world only ever accumulates, this buys you much less.
• Compressing context and caching context are in conflict. Every method that rewrites its prefix pays full price for it. Measure the bill, not the token count, and put your mutating block after whatever you want cached.
• A schema protects only against what its designer anticipated. A generic field to “write things down in” measured indistinguishable from no field at all.

The claims here are scoped to two models, one environment that discriminates, and a procedure short enough to sit below the caching threshold. What generalises is not any single number but the arithmetic: check that your effect fits inside the range your effect can have.

---

Replication of SKILL.state: Scalable Long-Horizon Agent Skills (Badhe, Tiwari and Chung, accepted at EMNLP). Related: The Scaffolding You Pay For on interventions that cost more than they buy, and The Forgetting You Don’t Measure on what a single benchmark number hides.

Correction, 13 September 2026: an earlier version of this article framed the retroactive-invalidation result as contradicting a limitation stated in the paper. It does not. The paper reserves that case for tasks whose objective is the history itself — auditing, explaining past actions — which this probe does not test, and its own Experiment 3 already reports the same direction qualitatively. The contribution is the metric, not the sign.