Every AI feature ships with a claim. The model understands your codebase, the profile personalises the output, the agent picks the right tool. Almost nobody checks whether the claimed thing is doing the work, or whether anything of the same size and shape would have produced the same result.
I built that control, ran it against my own product across 60 preregistered runs, and my product lost. The whole result is on this page.
For teams shipping a feature whose value depends on a model doing something specific, where nobody has yet run the version that should not work.
First three, free
If you can finish the sentence "it works because the model is doing X", X is what I test.
Not a fit if the feature has no claim that could turn out to be false, if you cannot run it more than a handful of times, or if the thing you want is a testimonial rather than a result. Send it anyway and I will say so in a line rather than leave you guessing.
A cold baseline tells you the feature changed something. It does not tell you the feature is what changed it. To learn that, the control has to be the same size, the same shape and the same tone as the real thing, and hold none of the substance.
Then the predictions and the pass rule get written down before the first run, so nobody gets to pick the metric after seeing the output.
Nothing at all. The baseline almost everyone stops at.
My mined working profile, 10,511 bytes, built from 1,843 sessions, 12,184 messages and 1.74M tokens of my own real work.
An invented editorial designer who moved into software. First person, confident, opinionated about typography and grids and motion, and deliberately silent on every dimension the run measured. It never mentions border radius, saturation, accent count or cards. It is not an inverted profile, because an inverted control guarantees its own result and proves nothing. Length matched at 10,545 bytes against 10,511, and the harness was set to refuse to start if that drift went past five per cent.
The question, the four tasks and why each one was in the set, the sample size, every metric definition, the predicted direction for each, the decision rule, and a list of the things I was forbidding myself from doing once results arrived.
An effect counts only if the two groups of five do not overlap at all. Every value in one beyond every value in the other.
For a prediction that names a direction, that happens by chance about once in 252 orderings. Anything overlapping is not an effect, however good the medians look.
A prediction counts as supported only if it separates on at least two of the three tasks both against cold and against the placebo.
60 runs, Claude Opus 5, three conditions, five runs per cell. All 60 completed, none failed, and all 45 generated pages loaded clean in a headless browser with no console errors and no failed requests, so the clause allowing a rerun on technical failure never fired and nothing was generated twice.
Not one of them separated the mined profile from the placebo.
| Prediction, stated in advance | Against cold | Against the placebo | Verdict |
|---|---|---|---|
| Fewer rounded corners | overlapping | overlapping | Inconclusive |
| Lower colour saturation | separated, 2 of 3 | overlapping | Unsupported |
| Fewer card containers | overlapping | overlapping | Inconclusive |
On the portrait landing page task the placebo produced 1, 1, 1, 1, 1 rounded corners. My profile produced 3, 4, 6, 7, 7.
That is separated, in the opposite of the direction I predicted, on a law my own profile states in as many words: big radii are banned on sight. The invented designer has never heard of border radius and beat me on my own rule. Normalise per thousand characters and the reversal gets sharper rather than going away.
Portrait landing page task, condition C against condition B, n=5 per cell, metric defined before the run
The most quoted number from the earlier version did not reproduce either. The terminal task had been 98 columns wide cold against 64 with the profile. At n=5 the profile and cold overlap almost completely, and the 64 belongs to the placebo.
The profile does change the output. What got no support is that the change is specific to what is inside it.
The saturation prediction did meet its leg against cold, separated, in the direction stated in advance, on two of three tasks. Leaving that out would be its own dishonesty. It is also not a refutation, and it is not a finding that a profile and a placebo are the same thing. It is the absence of evidence that the mined content is what did the work.
It is on this page because the correction is the method, and because the earlier numbers were quoted publicly before anybody checked them. Mine included.
You name the feature and what it promises the user. We write that down as something that could turn out to be false, because a claim that cannot fail cannot be tested.
I build the version that should not work. Same size, same shape, same confidence, none of the substance you are claiming. You see it and you get to object to it before anything runs.
Predictions, metric definitions, sample size and the pass rule, written down and sent to you before the first run, along with the list of things I am forbidding myself from doing once the numbers arrive.
Both arms, N times each, a fresh process every time, order randomised from a seed that gets recorded. Every transcript kept, including the ones the harness voids.
One line per prediction. Beat the control, tied, or lost, with the reason. Losses and ties stay in. You get the raw runs and the harness, so you can repeat the whole thing without me.
This is the part that should worry you, so here it is in front rather than in the small print.
You see the result before anyone else does, and you decide whether your name is on it.
The result gets published either way. That is what pays for the free ones. What you control is whether it is published as your company and your feature, or with both removed and only the shape of the result left.
You get the raw runs and the harness whichever way it goes, so you can rerun it, argue with it, or take it to your team.
And a loss found here is a loss found before your customers find it. Mine cost me a product page I had to rewrite. It was still cheaper than the alternative.
Nothing uploads from this page. Filling it in opens a draft in your own mail client with the answers in it, and you press send.
If no mail client opened, send this to founder@viberaven.dev
Nobody has bought this. The only product it has been run against is my own, and mine is the one that lost.