We scanned 2,783 public repositories for agent instructions that forbid something specific, then checked whether anything in their CI would actually stop it. Then we built the fix, and watched our own fix report success while it was broken.
Not whether a rules file exists. Whether anything would stop the rule being broken.
An AGENTS.md or CLAUDE.md that says "never edit this generated file by hand" is making a claim about the future. The interesting question is what happens on the day somebody does it anyway. If the answer is that lint, typecheck and the test suite all pass and the pull request merges, then it was never a rule. It was a preference, written down, hoping to be read.
So the scan had two halves. Find repositories whose agent instructions forbid touching a specific path, then read their published workflow files and ask whether any job mentions that path.
The rules themselves are ordinary and sensible. Do not edit the generated client. Do not hand edit the vendored icon set. Never commit the build artefact. Do not run the corpus script while another harness holds the binary. Every one of them was written by somebody who knew exactly what they were talking about, and not one of them was connected to anything that could say no.
Our first instinct was that a recently added prohibition matters more than an old one. Nobody writes "never touch this file" on a calm afternoon. They write it because something touched the file.
Six of the thirty two had been added in the previous sixty days. That looked like a signal until we read the commits that introduced them. Two of the six arrived in the same commit that created the instructions file, one of them called simply "add agent guidance". Those were not recent decisions. That was an entire ruleset adopted at once, and every line in a new file is new.
Rule age measures document age, not incident age.
The distinction that actually holds is whether the line arrived with the file or was added to a file that already existed. The second is somebody going back to a document they had already written in order to add a prohibition to it, which is more suggestive of learned experience, though a commit is not proof of an incident. Four of the six were added to a file that already existed. Of those four, two read like documentation work ("update agents.md to reduce LLM context") and two read like something happened. One named a specific trap and recorded that it had recurred twice.
Thirty two candidates, two worth acting on, and one of those belonged to a team we had already written to. We are reporting that number because a scan that only reports its hits is not a measurement.
The cheap version of enforcement is a CI job that fails a pull request which changes the protected path, unless a commit message says the change was deliberate.
The escape hatch is the part people leave out, and it decides whether the guard survives contact with a real team. A check with no legitimate way past it gets deleted the first time somebody genuinely needs to change the file, and a deleted guard protects nothing. This one does not forbid the change. It makes the change deliberate, and leaves the reason in the history.
It ships with a test that fails before the workflow exists and passes after, because a check that has only ever been seen passing has not been shown to work.
This is the part worth your time.
The generator had tests. The tests passed. The guard it produced was about to go out. Running the real thing against a scratch repository, by accident, with a base branch that did not resolve, produced this:
git diff had failed. Its output was empty. Empty was read as "nothing changed", so the guard reported success and exited zero. Any repository where the fetch hiccups, or the base reference resolves oddly, would have had every protected file silently unguarded, and the build would have stayed green the whole time.
A check that passes when its own measurement fails is not a check.
Two claims were being confused, and they are not the same claim. The generator was tested. The artefact it generates was not. Everything was green, and the thing actually being shipped had a silent failure mode nobody would have noticed until the day it mattered.
The repair is the two lines at the top of the fragment above. The reason this note exists rather than a clean success story is what the failure looked like from the outside, which was exactly like working.
| Case | Before | After |
|---|---|---|
| Pull request that does not touch the file | passes | passes |
| Hand edit of the protected file | passes | refused |
| The same edit, with the trailer | passes | passes |
| Base branch cannot be resolved | passes | refused |
Three of those four rows were already correct before the fix, which is precisely why it survived review. A guard is only interesting in the row where it says no.
Earlier this month we wrote to a company about a rule in their own instructions file, and suggested a CI step that fails when the protected thing differs from the base branch. The protected thing was a field inside a JSON document, not a file. You cannot ask git diff for a key.
The fix we described would not have worked. We found out because the generator refused to build it and said why, which is the only reason this paragraph exists rather than a quiet correction. A tool that will not produce an artefact for a case it cannot handle is worth more than one that always produces something.
The scan reads public repositories and public workflow files. A team may enforce a rule in a pre commit hook, a private pipeline, or a review habit that no public file reveals, and none of that would be visible here. "No CI job names this path" is an observation about what is published, not a verdict on anybody's discipline.
Two repositories could not be read at all, and are recorded as unreadable rather than as findings. Thirty two candidates out of 2,783 repositories is a hit rate near one percent. The honest summary is that this technique finds real problems and finds very few of them, which is worth saying plainly before anyone reads it as a product.
Companies are not named here. The examples above are real and the quotes are verbatim, and none of them is anybody's failure to publish. Ours is.
The same method. Your instructions files, your workflow files, and a written answer for each rule: whether anything currently stops it being broken, which ones are worth enforcing, and which ones cannot be enforced by a diff check at all.
Where a guard makes sense you get the workflow and the failing test, not a description of them. Where it does not, you get told that, with the reason.
A call first, so the scope is one real thing rather than a survey.
Book a call