Explicit State Doesn't Forget. It Misremembers.

In the first half of this replication I ran SKILL.state on Claude and found that its headline degradation did not appear there. The obvious objection is that the paper runs on Gemini-3-Flash, not Claude. So this half runs on theirs — gemini-3-flash-preview through Vertex, in an environment rebuilt to match their Appendix B — plus Claude Haiku 4.5 under the same environment and output cap, with every cell repeated and every episode kept as a per-step trace.

The short version: the paper’s central claim holds on its own model, more cleanly than the paper reports it. On a second model it does not, and the way it fails is the interesting part. Explicit state does not lose track of the procedure. It writes the state wrong, and nothing checks the write.

Repeat every cell, even at temperature zero

The protocol started from an assumption that sounds safe: with greedy decoding, a seed is an instance of the environment, and one run per cell is enough. It is not. The same cell, repeated five times at temperature=0, scored anywhere between 0.830 and 0.960. The distribution has a long left tail, because failures cascade: the agent stores one pallet on the wrong shelf, and every later decision that touches that shelf inherits the error.

Three conclusions written up from single runs — an environment effect, a monotonic noise curve, a noise estimate — did not survive repetition. Every figure below is a mean over 15 to 24 runs, with the spread computed over runs, not over seed means.

On their model, the thesis holds

Their Table 1 has four runtimes: ReAct (full transcript), Memory (rolling summary), Stateful (state block plus transcript) and SKILL.state (state block only). Here are the two that carry the argument, ours against theirs:

SKILL.state does not fail once in 75 episodes of up to 200 steps, where theirs loses six points. The direction of their thesis reproduces with room to spare. The magnitude does not: our ReAct loses 0.080 between 10 and 200 steps where theirs loses 0.160, and sits 17 points above theirs at T=200. Reasoning budget, output cap and the three environment gaps I could find against their appendix were each varied, and none moves it. The model name is theirs, but gemini-3-flash-preview may not be the exact checkpoint behind their Gemini-3-Flash, so I cannot rule the model out — only say I did not change it.

Memory is the one arm where the comparison is between two different things. The paper does not specify how its Memory summarises, so a replication has to pick a policy, and ours keeps a much tighter summary: at T=200 its mean prompt is a fourteenth of theirs. With that tighter summary Memory is the arm that degrades most, last at T=100 and T=200. How much of that belongs to summarisation as such, and how much to how hard it compresses, these runs cannot say.

On a second model, explicit state writes the state wrong

Same environment, same output cap, Claude Haiku 4.5 through Microsoft Foundry, 24 runs per cell at the two ends of the horizon:

On Haiku the ordering turns over. The full-transcript arm is almost flat (−0.005 from T=50 to T=200) and SKILL.state is the one that drops. To find out why, I replayed every episode against the real environment and compared, step by step, the inventory the model believes — the one in its state object — with the one that is actually on the shelves.

Every failure of explicit state has the same origin. In all 17 episodes where the belief drifted from reality, the step that corrupted it was a Move executed correctly with a state patch written wrong. Move is the only transition that makes the model copy one shelf’s contents into another key of its own state, and that copy is where it slips.

It is systematic, not noise: in 6 of the 8 runs of seed 2 the corruption happens at exactly that step, and from there the agent cascades through about 25 failures. Not every miswritten patch matters — in seed 0 the model copies the wrong lot number, which no later decision reads, and those runs lose nothing — but the write is never checked against what the action did. Gemini never made this error in 75 episodes; Haiku wrote at least one wrong patch in 17 of 24 at T=200.

The paper does study erroneous state updates — premature overwrites, schema and type errors. This is a narrower thing: a schema-valid patch with the wrong value in it, which passes every check the runtime runs.

Keeping the transcript absorbs it

The Stateful arm is the control that makes this legible. It maintains the same kind of state block, and it also keeps the full transcript. On Haiku it makes the same kind of write error — its belief drifts from reality in 8 of 15 episodes at T=200, always a correct action with a miswritten patch — and loses nothing to it: the wrong state prescribes a different action on only 4 steps, and on all 4 the model does what reality requires. None of its failures comes from its state.

On Gemini the same arm behaves the other way round: its state drifts in 11 of 15 episodes, where SKILL.state on the same model never drifted, and 77 of its 211 failures happen with a correct state in front of it. Stateful and SKILL.state also differ in response format, parsing and retries, so this describes rather than isolates the effect of the transcript. But the practical reading is hard to avoid: on the model that misremembers, the redundant copy of the history is what catches it.

Where explicit state wins, measured by the decision

The paper’s recovery experiment asks what happens when a fact the agent was told stops being true. I measure it per episode: on a correct trajectory the correction decides exactly one step in each of the three scenarios used, so the question is whether the agent gets that step right. Three models now, the same design on each:

Explicit state applies the correction in 70 of 72 episodes (Wilson 95 % interval 90–99 %); the full transcript in 10 of 72 (8–24 %). This is the paper’s own claim, and it reproduces cleanly. It is also smaller than the “93 of 93” I reported in the first half: that count read its dependent steps off a simulated trajectory instead of the agent’s real one, and it could not register a miss that came from the agent doing nothing. Measured on the real trajectory, the direction survives and the perfection does not.

The other probe tests the limitation the paper declares: explicit state only protects what its schema anticipated. A fact announced at step t first becomes relevant at step t+40, and the agent either stored it somewhere or did not.

Sonnet 5 is left out of this table because a quarter of its episodes never reach the test — its trajectory has diverged by step t+40 — and its cells need two rates to read honestly; they are in the paper.

A place to put it is not enough; the place has to say what goes there. “No dedicated field” does not mean nowhere to store it — Sonnet’s few successes in that arm wrote the quarantine straight into the inventory object — but the named field is what reliably works, and these experiments cannot separate the extra slot from the cue its name gives. For the runtime with no schema at all, a reminder attached to the observation does the job: ReAct goes from 0–34 % to 71–96 % across the three models.

The bill

The cost half of the first article stands. With a short procedure, caching shrinks SKILL.state’s input-cost advantage over the transcript from 7.5x to 1.4x on Anthropic, because an append-only transcript is an ideal cacheable prefix and a mutating state block is not. On Vertex the same accounting barely moves the ratio — 7.5x in tokens is 7.2x in effective input — because implicit caching there saved ReAct 6.8 % of its input, against 82 % on Anthropic.

And prompt order, isolated this time, is a first-order cost variable. The same Stateful runtime, sending the same content, with the transcript marked as a cacheable prefix in both arms, costs 5.2x more when its state block goes in front of the transcript than behind it — 869k against 168k effective input tokens per 50-step episode — for the same score. The paper’s own prompt template puts the state block in front.

What I would take from it

• Explicit state does what the paper says on the paper’s model. On Gemini it never loses a step, and it applies retractions almost every time on all three models.
• Its failure mode is writing, not remembering. A schema can validate the shape of a patch; nothing in the runtime checks that the value matches what the action did. If you build on explicit state, that check is the part to add.
• Keeping the transcript next to the state is cheap insurance on some models. It cost Stateful nothing on Haiku and caught every wrong write that would have mattered.
• Put the mutating part last. It is a one-line change and, measured in isolation, a 5.2x difference on the invoice.
• Repeat the cell. Temperature zero is not a reproducibility guarantee, and a single run hides exactly the tail where the failures live.

Every number here was recomputed from the per-step traces by an independent adversarial reviewer — nine rounds of it, each allowed to read the raw data and none of the prose — and the paper, the traces and the code are public.

---

Code, per-step traces and the full paper draft: JaviMaligno/delayed-relevance. The original paper: SKILL.state (arXiv 2608.26263). The first half of the replication: When the Fact Stops Being True.