We run autonomous agents that send real email to real companies and write to a real ledger. This is not a list of what they do well. It is what they got wrong, what it cost, and the guard that now catches each one.
The useful question to ask a vendor is not whether their agents work. It is what happened the last time one did not, and whether anybody noticed.
Every failure below is real and dated, and the test that now prevents it is in the repository. None was found by a demo. All of them surfaced while the system ran against the actual internet, and two were found by a reviewer that had not written the code.
What the system claimed. The message was undeliverable, and the address belonged to a colleague named in the body.
What actually happened. A person replied. Their subject line happened to contain the word undeliverable. The classifier matched the word, called it a bounce, attributed it to a different human mentioned further down, and never wrote it to the funnel at all.
Business consequence. The one thing the entire system exists to detect. A human answered and the machine threw it away.
Root cause, and this is the uncomfortable part. Four bounce-detection defects had already been fixed by the author. This was the fifth, and the author's own tests could not have caught it. The fixture built a bounce body with exactly one address and no quoted original, while every real bounce carries at least two. The ritual of breaking a test to prove it works had been passing against a fixture incapable of failing.
Guard. A bounce now requires a daemon sender or a genuine DSN part. Word matching cannot classify anything.
How the guard was tested. Two of the author's own tests were deleted, because they encoded the wrong behaviour and would have defended the bug. The finding itself came from an independently prompted reviewer that had not written the code.
Fixtures must be built from captured reality, not from what the author imagines the input looks like.
What the system claimed. Three courteous first-contact messages, sent successfully.
What actually happened. Each one closed with no thanks is a complete answer and I will not follow up. Written as manners. It is a promise, and the follow-up sequence would have broken it on the second touch.
Business consequence. Three recipients permanently suppressed. Roughly a fifth of the first cohort, thrown away by one sentence nobody had thought of as a commitment.
Root cause. Copy treated as decoration rather than as a constraint the system would later have to honour.
Guard, and this is the design choice worth stealing. The sentence is not banned. It is good copy and it lowers the bar to replying. Instead the sender detects an unconditional no-follow-up promise and writes that recipient into an append-only suppression file that the follow-up guard reads. Writing the promise makes it true. A conditional opt-out is correctly not treated as a promise, which matters, because every message carries one.
How the guard was tested. On 22 August a different session ran a fifteen-message follow-up batch with nobody supervising it. All three promised addresses were absent from it. The guard held in a batch no human was watching, which is the only test that counts.
When the system can make a statement true, prefer that to forbidding the statement. Bans get worked around. Enforcement turns the claim into a fact.
What the system claimed. Submitted successfully. The page said so.
What actually happened. The text thanks, submitted successfully was set to display:none, occupied zero pixels, and belonged to a newsletter signup elsewhere on the page. The agent matched the words and reported a win.
Business consequence. A prospect recorded as contacted who was never contacted. A false entry in the one ledger the whole operation is measured on.
Root cause. Verification by text. Text is the cheapest thing on a page to get wrong.
Guard. A submission is confirmed by state change plus transport evidence, never by matching a string. Either the network shows the request and its status, or the form element is gone from the page.
How the guard was tested, and this one fired again today. A form was filled and submitted for a real prospect. The page showed an ordinary failure notice, and the network log gave the reason: POST /api/contact returned 404. Their own endpoint does not exist, so every application through that page fails silently, for everyone. Nothing was recorded as sent, because nothing was sent. Under the old behaviour this would have been logged as a successful contact.
Verify an effect by the mechanism that produces it, never by the message that describes it.
What the system claimed. Held jurisdictions were being respected.
What actually happened. The rule never contact the same recipient twice was enforced in code, and held perfectly. The rule hold these jurisdictions lived only in agent instructions. When the hold list turned out to be wider than assumed, we could not answer did we already mail one of these from our own records, because no send had ever stored the jurisdiction it went out under. One had.
Business consequence. One message into an opt-in jurisdiction that cannot be unsent, and an audit question with no answer available anywhere.
Root cause. Compliance state carried in prose rather than in the write path. An agent can forget its prompt. A function signature cannot.
Guard. The sender refuses any message carrying no jurisdiction at all, and refuses a held jurisdiction by name, re-checked at the moment of writing rather than at the start of the run.
How the guard was tested. Six tests, two verified by mutation. The guard was deliberately broken to confirm the correct test failed for the correct reason, then the file was restored and checksummed to prove nothing had been left behind.
If a rule has legal or financial consequence it belongs at the write boundary, not in the instructions of the thing being constrained.
What the system claimed. A portfolio page, about to be offered to a prospect as proof of craft, stating everything below is 1920x1080 at 60fps.
What actually happened. Measured with ffprobe before the message went out. Two of the three films were 1280x720 and the third was 1152x648. All three sat far below the delivery bitrate the same business promises its clients. The masters were genuinely 1080p60; the web copies had been downscaled at some point and the sentence was never updated.
Business consequence. Came close to sending a motion studio a page that overclaimed the one specification anyone in that trade checks first. It had been wrong for weeks. Nobody would have noticed for months.
Root cause. A claim written once, beside files that changed later. Nothing tied the sentence to the artefact it described.
Guard. Any number in outward-facing copy is measured against the file it describes before that copy is used as evidence. The films were re-encoded from master at 1080p60 with colour explicitly tagged, and the one claim that could not be made true, on a film with no 1080p master, was removed rather than softened.
How the guard was tested. Both pages were re-measured on the live site after deploying, not on the local copy. While measuring, the primary button was found to be 37 pixels tall against a 44 pixel minimum, on a page whose entire purpose is being tapped by a thumb.
An agent that cites evidence has to check the evidence, or it launders a stale claim into a confident one.
A vendor telling you their agents are reliable is worth nothing. So we built a crash test that attacks our own system across twelve named failure classes, and ran it.
Each class is attacked for real rather than described: fire under a grant that was never given, spend past a daily cap, record a send with no receipt from the server, contact the same recipient twice, kill a job mid-run and resume it, abandon a job in the running state, cross a day boundary, read a record written before a schema change, name unauthorised access in a plan, request an unbounded time budget, remove the authority file entirely, and reconcile work that produced output but died before marking it.
The first run returned twelve of twelve passing. That is not a result worth publishing. A suite that only confirms its author is a marketing document, so we wrote a second adversarial round chosen for the opposite reason: each attack picked because the guard was plausibly wrong.
Real defect one, and the serious one. Eight threads writing at the same instant against a cap of one produced six fires. The cap was checked, then the ledger was appended, as two separate steps, so any number of processes could pass the check together. More than one session runs against this ledger, which makes it reachable rather than theoretical. A cap that is only checked before the write is not a cap. Both steps now happen inside one lock. The same attack now produces exactly one fire.
Real defect two, and it was invisible by design. The rule that holds certain jurisdictions matched by plain substring, so United States. Founder is originally from Germany. was blocked. A false positive here never announces itself. It silently deletes a lawful lead and looks exactly like the guard working correctly, so nobody ever investigates it. The rule now matches the buyer's own jurisdiction, the first clause, on a word boundary.
And three times the crash test was simply wrong. Twice in the first round it reported a problem that was a bug in the test: it passed a date where a datetime was required, and it called a deliberately time-gated staleness check one second after creating a job, then reported the absence of a guard that was working correctly. The third was worse, because it looked convincing: it reported that the never-contact-twice guard was case sensitive. It is not. The guard lowercases when it reads. The test had asserted on the shape of the stored row instead of on the decision the guard actually makes.
Testing the store instead of the decision is how a crash test invents a bug that was never there.
Every fix carries a regression test, and each regression test was verified by breaking the fix on purpose, confirming the correct test failed for the correct reason, and restoring the file with a matching checksum. Removing the lock fails the concurrency test. Restoring it passes. The file is byte-identical afterwards.
The scorecard today is sixteen of sixteen. That number only means something because of the three false findings and the two real ones behind it. A clean first run would have meant the attacks were too gentle.
These were not chosen to look humble. Four of the five were caught by a guard or a reviewer rather than by anyone reading carefully, and the fifth was caught by a measurement taken only because a claim was about to be used as evidence.
The shape is identical across all five. Every one was a system reporting success it had not verified. A bounce it had not confirmed. A promise it had not recorded. A submission it had not observed. A jurisdiction it had not stored. A resolution it had not measured. A cap it had checked but not held.
If you are deciding whether to let an agent post an invoice or email your supplier, the question is not whether the vendor's agents fail. They will. It is whether the vendor can show you the last five times, what each one cost, and the test that now catches it.