Ask the agent to refactor. Writing the rule isn't enough

Codex gets a task in a budgeting app: schedules whose amount “is between” two values should now be planned at the worst end of the range, not the middle. It searches, writes “I found three midpoint calculations”, moves all three into one shared function, runs the tests, and reports: “Previews, automatic posts, and synced rule actions all use the same calculation.” The tests pass. The step fails.

There was a fourth copy. The budget template computed the same midpoint, written differently (Math.round(num1 + num2) / 2), and Codex’s search never matched it. In five of eight chains it never opened that file, and in those five it missed the copy every time.

This is the failure that abstraction is supposed to prevent. If the rule lives in one place, there is no fourth copy to miss. Matt Pocock put the case for more abstractions in a tweet that I have been testing for two weeks:

My hot take from chatting to @poteto is that we should use MORE abstractions in the AI age. You can use them (combined with harsh lint rules) to reduce the design space available to the agent and constrain them only to good decisions. […] Be braver! Design abstractions.

He is talking to the humans who design the code. My question in this third round was the next one: if the code is already repeated, can the agent be the one who designs the abstraction? And what makes it do so: a rule, a lint, or asking?

Where this came from

I ran this in three studies, all on the same real codebase: loot-core, the core of the open-source budgeting app Actual, about 48k lines of TypeScript. The agents were Claude Code with Sonnet 5.5 and with Haiku 5.5, and Codex with GPT-6 Luna, all at medium effort.

The first study tested the “harsh lint” half directly. With the habitual code, the agents made zero bad design decisions in 240 runs each, so neither more abstraction nor lint had anything left to remove. Nor did they save tokens.

The second study built two versions of the same four areas of code: one where the logic is repeated in several places (R), and one where a human had factored it into shared functions (F). Then it ran chains of four tasks on each. With F the agents missed fewer copies and blind judges scored the design higher, especially across a whole chain: Sonnet’s chains scored 2.94 out of 5 in R against 3.56 in F. Token cost did not change. So abstraction helps, but in that study a human had designed it.

The experiment

Every arm starts from the repeated code of study 2 and runs the same chains. Each chain has four steps, each in a fresh session, so whatever an agent learns can only reach the next step through the code. T1 changes a business rule, T2 adds a new consumer of that logic, T3 changes the rule again, T4 builds something on top. The arms differ in one lever:

• RG, the written rule. One line in the root AGENTS.md, worded as a team convention: “Do not copy logic: when new code needs behaviour that already exists, extract it into a shared function and reuse it; when a rule changes, change it in one place.”
• RL, the harsh lint. yarn lint runs a duplication check (jscpd) that fails if any clone touches lines the step added. Removing unrelated duplication elsewhere earns no slack.
• RC, the directed refactor. Before T1, a step T0 that names the behaviour area and asks for it to live in one place, with behaviour unchanged. It doesn’t say which files, how many copies, or where.

That makes 384 chains and 1,632 steps. Correctness comes from hidden tests. Design comes from blind judges (Claude Opus 5.5), who scored 2,206 diffs without knowing the arm, the agent, or that arms existed. A second judge scored 340 of them, and I computed agreement before comparing anything: weighted kappa between 0.70 and 0.89 on every score. The design and every analysis were registered before the runs.

Only one lever reaches all three agents

The main measure is the maintainability of a chain’s full diff, as the judges scored it from 1 to 5:

For Sonnet, the three levers all work, and with the registered judge the rule and the lint survive correction for multiple tests. For Codex and Haiku, the rule and the lint do nothing measurable. Only the refactor request moves all three. On its own, each agent’s refactor effect is around p = 0.02 and would not survive correction, but it points the same way in 11 of the 12 code areas, and it is the only lever that also moves correctness.

Haiku is a special case: it starts close to the ceiling (3.56 against 3.91 for the human-factored code), so there is little room to gain.

The judges are Claude, and so are two of the three agents. To check that they weren’t just preferring their own family’s code, I had GPT-6.1 Sol, through Codex, re-score the 480 chain diffs blind with the same rubric. The two judges agree well on maintainability (weighted kappa 0.79), and there is a family effect: compared with Sol, Opus scores Claude-written code 0.21 points higher than Codex-written code. Every comparison above is within the same agent, so a constant offset cancels out, and with Sol as the judge the refactor request still raises all three agents (+0.34, +0.25 and +0.28 points; p = 0.03, 0.09 and 0.04). The written rule still helps Sonnet (p = 0.02). What shrinks is the lint’s gain for Sonnet, from 1.13 to 0.43 points in the areas where it can see copies: the family gap is largest in the rule and lint arms.

What the refactor buys: search

Back to the missed copy. With the repeated code and no lever, Codex misses a copy at a rule step in 33% of cases and gets 59% of steps right. After its own refactor, it misses 12% and gets 70% right. That is better than the code a human had factored in study 2 (22% and 66%).

When I read the traces, the mechanism was simpler than “the agent designed a good abstraction”. In the repeated code, Codex does consolidate at the rule step, but only the copies it finds. The refactor request says the logic is “repeated rather than defined once”, and that sentence sends it looking for every copy. A rule change doesn’t. Once the copies sit behind one function, the next rule change reaches all of them with one edit, including the budget template, without the agent ever opening that file.

The two misses that remain repeat, with Codex’s own abstraction, the two failures I had seen with the human one. In one chain, the refactor never found the template copy. In the other, Codex had preserved the template’s odd rounding as a named mode, and at the next step it treated that mode as a protected requirement: “Schedule templates explicitly request a separate sum-first rounding mode; I’ll preserve that behavior.” An abstraction that names an exception invites the next agent to protect it.

All three agents built the same kinds of abstraction at T0: a category-selection function, one shared flow for linking bank accounts, one weekend-rule function. The judges found most refactors complete: the median share of copy sites routed through the shared definition was 100% for every agent. Behaviour was kept too: my behaviour check flagged four of the 96 refactors, and only two of those were real changes (both Codex’s). The other two were an artefact of how finely the check records internal calls.

The lint, and how Codex got past it

With Sonnet, the duplication check produced the most clearly better design in the study. In the bank-sync area it built one linking pipeline for every provider. A judge wrote: “Adding a provider becomes a few lines.”

With Codex it produced something else. In four steps it marked the code so the checker would skip it:

// jscpd:ignore-start -- This provider has the same link lifecycle as the others.

with the justification “The duplication check is unusually strict here: it flags the same established provider lifecycle even when the implementation is intentionally parallel.” In other chains it reused by stretching another provider’s function with flags until it served both. A judge described one: “a SimpleFin handler stretched with an ‘orgDomain’ in-check to also serve Fintegra.” In others it kept copies reshaped enough that the checker stopped matching them. The duplication count went down, and the design scores didn’t move.

The lint also has a structural blind spot: it only sees copies that look alike. In the area where the duplication is four different implementations of the same weekend rule, it never fires, and Sonnet’s design there came out slightly worse than with no lever.

What not to oversell

When I read the judges’ justifications, most of the design gain is fewer copies of the same rule rather than a different structure. The main complaint about the repeated code, “this rule lives in several places”, is still there in every arm. What changes is that six places become two. If I account for the number of duplicated blocks the judge itself counted, the arms’ effect on Sonnet shrinks to between a fifth and a half. The one-pipeline case is the exception.

The refactor doesn’t pay for itself in tokens over four tasks. With T0 included, a refactor chain costs 16% to 23% more than one with no lever. A longer chain might recover it. This one didn’t.

The written rule is never quoted. Sonnet follows it without citing it, and it does so mostly by reusing helper functions it created itself in the previous step. Codex has the file in context, and it follows another instruction in it (run commands from the repository root) in 541 of 544 steps. This one it ignores.

What I would do

If you want an abstraction in an agent-maintained codebase, ask the agent for it as a task, not as a rule. Name the area, say it should live in one place, and say which differences must survive. The last part matters: an exception that is not named either gets erased or gets wrapped in a mode that the next agent treats as sacred.

A rule in AGENTS.md costs nothing, but whether it works depends on the agent. A duplication lint is a safety net, not a designer: some agents meet it by building the abstraction, others by getting past the check. And none of these levers replaces the search. What made the difference was that someone asked the agent to go find every copy.

Pocock asked people to be braver and design abstractions. My data adds one thing: when the code is already repeated, you can delegate the design, as long as you ask for it explicitly.

---

Limits: one codebase, one effort level, three agents. The registered judge and its second pass are the same model; the cross-family check with Sol covers the chain diffs, not every score. With four code areas, the per-area sign test can never reach significance. A related result: in an earlier experiment on how the way code is written affects agents, the effects only appeared in large repositories. The pre-registered design, the code, the anonymised data and the full report are in the repository.