Agent Readiness Audit — my own system

Evidence: 21 experiments to date. 13 reverted automatically. 1 kept. 3 escalated to me. 4 still measuring.

The finding worth stealing: six of the thirteen reverts had improved the number they were aiming at. One raised its target axis by 8.33 points — and was reverted, because it dropped a different axis by 2.67. Three more improved their target and regressed two or three others. The rest improved their target but not by enough to clear the bar.

Three further changes were reverted for a reason I'd recommend to anyone: they couldn't be measured inside the window. Not “measured and found wanting.” Unmeasurable. The rule is that unmeasurable means revert, and it is the single highest-value rule in this system. Without it, every change that resists measurement accumulates permanently.

The one change that survived raised its axis by +0.87 and regressed nothing. A 1-in-21 keep rate is not a sign the loop is broken. It's the number I'd expect from an honest gate, and it's the reason I trust this workflow more than anything else I run.

Verdict: safe, and the one point it's missing is honest. The tests that exist are good; they don't cover everything. I'd rather report a 1 than round up.

Supporting evidence: a separate adjudication ledger shows 25 decisions on proposed changes — 14 approved, 10 rejected, 1 applied. A gate that never rejects anything isn't a gate. Ten rejections is what “real” looks like.

What went wrong: the function that found the next open publishing slot looked fourteen days ahead. One channel gets eight slots per fourteen days. Once those were claimed, the function returned “nothing available” — and every caller treated that the same way it treated “nothing to do.” Silently. Skip.

16 approved posts sat invisible for up to 30 days. A repair job that ran on a schedule re-skipped them on every single pass, for a month, without logging that it had.

Two things here are worth more than the bug:

The check existed one layer too early. Content quality was measured carefully, on named axes, with real thresholds. Nothing measured whether an approved post reached a schedule. The system was rigorously verifying the part that was already fine.

The human gate made it worse. I approved those sixteen posts. Approval felt like handling, so it replaced the question “did this actually ship?” instead of prompting it. A human in the loop raises confidence whether or not it raises verification — and when the two come apart, the gate is worse than no gate, because it retires your suspicion.

What went wrong: a file was renamed between the moment it was noticed and the moment it was read — routine behaviour for editors and sync tools, which write to a temporary name and rename into place. The resulting error escaped into the dispatch layer and killed the observer thread.

The supervising process kept reporting the unit healthy while it observed nothing, for as long as it stayed up.

That sentence is the whole reason I score Verifiable separately from Recoverable. The process was alive. Alive was the metric. Alive was true, and meaningless.

The rule: liveness is not an output metric. If the only thing your monitoring can tell you is that a thing is running, you are not monitoring it. Count what it produced in the last hour and alert on zero.

What went wrong: matches were scored by cosine similarity against an absolute threshold. Sentence embeddings place any two English technical sentences at roughly 0.75–0.85 similarity regardless of topic. The threshold had been set to 0.25 back when everything was scoring zero — so instead of matching nothing, it matched everything.

On one briefing: all 20 items matched. Scores ranged 0.763 to 0.836. The margin between the winning goal and the runner-up ran from 0.0004 to 0.033, median 0.0107.

Winners were being decided by floating-point noise, and every one of them carried a confident-looking score. A weather-forecasting item was matched to threat-intelligence escalation. It looked exactly as plausible on the dashboard as a correct match, because the number was in the same range.

Requiring the top goal to beat the runner-up by a real separation took that briefing from 20 of 20 false matches to 0 of 20.

Verdict: this should never have been running unattended. A 0 on Verifiable and a 0 on Delegatable is the exact combination the scorecard exists to catch, and I shipped it anyway — because it produced output that looked right, every day, for months. Fixed 2026-08-07.

The rule: a similarity score is not a confidence score. If a ranking system can't tell you how much the winner beat the runner-up, it isn't ranking. Threshold on the margin, not the magnitude.

If you want to score your own workflows against the same five gates, the Scorecard is free and runs entirely in your browser — nothing you type leaves the page.

If you'd rather I ran the whole thing on your workflows — every candidate scored, the ones you graded generously dug into, a ranked fix-list you own — that's the Audit. Two weeks, fixed scope.