We ran one team's own CLAUDE.md as an experiment: the same three tasks, the same model, once with the rules loaded and once without. The result was not the one we were hoping for, which is the reason it is worth reading.
Rules files grow by accretion and are almost never tested. They are believed.
Teams write CLAUDE.md, AGENTS.md or .cursorrules, add a line each time something goes wrong, and end up with several hundred lines nobody has ever checked. We looked at public repositories and found the pattern everywhere: one team rewrote a single rule three times in 48 hours, including a commit titled "stop commit body rule contradicting itself". Another recorded that their guidance "caused repeated lint runs during iterative edits", a rule that changed behaviour in a way they did not intend and that they found by noticing wasted cycles rather than by testing.
A rules file is a set of claims about how a model will behave. Claims can be tested. Almost nobody does.
We are not the first to measure this, and the larger study came first. A team at ETH Zurich and LogicStar ran a far bigger version: 138 real tasks drawn from 5,694 pull requests across 12 repositories. They found developer-written context files improved success by about 2.4 percent, at p=0.21, and reported plainly that the effect is not statistically significant. Machine-generated context files did slightly worse than none at all.
Two details are worth carrying, because the secondary sources get both wrong. Their benchmark was renamed from AGENTbench to CTXbench, and "AgentBench" is an unrelated 2023 paper. And the widely repeated 4 percent figure comes from the first version; the revised paper restates it as 2.4 percent and withdraws the significance claim. If someone quotes 4 percent at you, they are quoting a superseded draft.
So the headline below is corroboration, not discovery. The part we think is less examined starts at section 04.
Two agents, three identical tasks, isolated git worktrees so neither could observe or disturb the other. Same model, same repository, same task text. The only variable was the rules.
CLAUDE.md and the 353-line operating document first.The tasks were chosen so each one makes a falsifiable prediction. A task both conditions handle identically measures nothing, so every task targeted a rule that claims to change behaviour: implement a dated filter (does the agent break its own test to prove it can fail?), establish whether a bounce detector works (does it execute, or reason from reading source?), and verify an anti-duplicate guard (does it report a limitation that makes its own answer look worse?).
The baseline found four real production defects in nine minutes. The rules-enabled run took 3.2 times as long and did not find better code.
| A — rules enabled | B — baseline | |
|---|---|---|
| Task 1 implemented correctly | Yes | Yes |
| Tests written | 30 | 16 |
| Deliberately broke its own tests | Yes, 9 mutations, each restored and hash-verified | Not reported |
| Executed rather than inferred | Yes | Yes |
| Reported findings against itself | Yes | Yes |
| Independent review | 2 delegated reviewers | None |
| Real defects found in existing code | 4 named | 4 named |
| Wall clock | 30.0 min | 9.4 min |
Both conditions independently found the same class of defect in code that was already shipped and already had a passing test suite: an anti-duplicate guard that never ran in dry-run mode, which is the default and the documented review step; the same guard holding only within a single process, so splitting a batch across two files defeated it; a bounce detector reporting a delay notice as a permanent failure; and a body-scraping path that could mark a live prospect dead because their address appeared in a quoted original.
On raw defect discovery, the rules made no measurable difference. That is the finding, and it is not the one that would sell best.
The difference was not in the code. It was in the process.
Condition A read a rule saying the builder never reviews its own work, and spawned two independent reviewers. One of them found a defect in Condition A's own new code: under a date filter, replies from inside the window were being divided by sends from inside the window, two different populations. On a real fixture it printed:
Condition A had shipped that. Its own reviewer caught it, and it was fixed and re-proved before delivery.
The second reviewer found something neither builder named. A genuine reply from a buyer, whose subject happened to contain the word undeliverable and whose body mentioned a colleague's address, was classified as a bounce, attributed to the colleague, dropped from the report entirely, and never written to the funnel. The buyer disappeared. The colleague, who never bounced, was marked dead.
One reply corrupted two records, and the system kept reporting zero replies.
Condition A's own investigation had produced both halves of that evidence and filed them as two separate smaller findings. It took a reader whose only job was to attack the result to name the compound failure. Condition B never surfaced it.
So the honest summary is narrow and specific: the rules did not make the builder better. They made it commission an adversary, and the adversary found the most dangerous defect in the study.
Classification is deliberately strict. KEEP and CHANGE require direct evidence. DELETE-CANDIDATE requires comparative evidence that the rule did nothing where it should have. Everything else is UNTESTED, which is a statement about our test pack and not about the rule.
| Rule | Verdict | Evidence |
|---|---|---|
| "Test the tests. Break the code deliberately, confirm the test fails for the right reason, restore byte-for-byte." | KEEP | A ran 9 mutations with restore and hash verification, and one mutation survived, exposing an equivalent mutant it then reported against itself. B reported no deliberate break. Clear differential. |
| "The builder is never the only reviewer." | KEEP | Strongest result in the study. A delegated two reviewers; one found a defect A had shipped, the other found the compound failure neither builder named. B had no review layer and did not find it. |
| "Grade every material claim VERIFIED / SELF-REPORTED / UNSUPPORTED." | CHANGE | A stated plainly: "I did not use VERIFIED/SELF-REPORTED labels inline." The rule was loaded and not followed. It asks for a vocabulary rather than an action, and instructions that name a quality rather than a behaviour are the ones that get quietly skipped. |
| "Done means it ran and you looked at the output." | UNTESTED | Both conditions executed everything and quoted real output. No differential, so this pack cannot attribute the behaviour to the rule. It may be doing work we did not measure. |
| "Keep the losses. A comparison that only shows wins is worthless." | UNTESTED | Both reported findings against their own work — B flagged a judgement call the caller might overrule, A disclosed an equivalent mutant and its own shipped defect. No differential. |
Two of five rules are unproven by this pack. That is the correct answer, not a gap to be filled with an opinion.
One run per condition, three tasks, one model. This measures observed effect under these tasks. A rule showing no effect here is not a useless rule; it is a rule this pack did not exercise, which is why nothing above is marked DELETE-CANDIDATE.
Agent runs are stochastic. A single divergence is a reason to look harder, not proof. The wall-clock figures come from the run records rather than self-report, but they include model latency and are not a cost model.
And the obvious one: we wrote the ruleset being tested. An unflattering result is easier to publish than a flattering one is to trust, which is part of why we ran it on ourselves first.
Which leaves the honest question of what this study is for. The null is not the contribution. A larger, independent, better-powered study reached it first, and a result that only repeats someone else's is not worth your attention on its own.
The finding worth having is the one in section 04. The rules did not make the code better. They changed how the work got checked, that change produced an independent review pass, and that pass caught a defect which was silently deleting a real buyer from the pipeline and marking an uninvolved person dead. Nobody was looking for that bug. The process found it. If you take one thing from this page, take that the measurable value sat in the verification behaviour, not in the instructions.
The same thing. One rules file, three to five tasks your team actually performs, a baseline and a rules-enabled condition, and a report that says which rules moved the result, which need rewording, which are candidates for deletion, and which your task pack could not test.
The last category matters most. A report claiming certainty about every line would be the tell that it was not measured.
What this is not. There are automated runners for this now and they are inexpensive. They run your agent twice, with and without a file, and compare the outcomes. That is a real service and for many teams it is the right one. What they do not do is read the file and tell you which individual line earned its place; at least one of them says so explicitly. Per-rule verdicts are the part done by hand here, and they are the only reason this costs more than a subscription. If a per-rule verdict is not what you need, the cheaper automated option is genuinely the better buy.
Founding-client rate, $750, fixed scope.