$ cd ../blog

The agent says it's done. Here's how I check

The most dangerous thing a coding agent produces is a good-looking report. All 1,033 tests passed, the linter was clean, the work was done. Every part of that report was true, and the build still failed in CI.

Agents don’t lie on purpose. They report what they observed, and what they observed is often narrower than what you need to know. Since all of my code is written by agents, checking their work is most of my job now. Every check below exists because something got through without it.

A fresh reviewer, until it finds nothing

The author of a change is the worst reviewer of it, and that’s as true for agents as for people. An agent that wrote the code remembers why it made each choice and defends it.

So when I’m in the loop, every change gets reviewed by a new agent that sees only the diff. If it finds a problem, the fix goes back to the agent that wrote the code, because that one knows the context. Then a different fresh reviewer looks again. The loop ends when a reviewer says “No issues found”, not when the first one is satisfied with the fixes it asked for.

It takes longer than you’d expect. One Pressa feature, a preview for documents over the plan’s size limit, went through six review rounds: two FAIL verdicts, fixes, and then a PASS. Another review caught a bug no test covered. A User-Agent header with invalid UTF-8 would make a database insert fail inside a rescue block that swallowed the error, so the request would get its PDF and never count against the monthly quota. Rails hands you headers as binary strings, and Ruby’s valid_encoding? returns true for a binary string even when the bytes aren’t valid UTF-8, so the obvious guard doesn’t guard anything. The fix scrubs every header before it’s stored, and the regression tests fail on the old code.

Run it the way the real environment runs it

In April, two Pressa pull requests passed 1,033 of 1,033 tests in my local Docker setup and failed in CI. The difference was one setting: locally the test environment loaded code lazily, in CI it loaded everything at boot. Under eager loading, Rails refused a change to the autoload paths and a file whose name didn’t match the class it defined. Lazy loading never looked at either.

The agent’s report was accurate. It ran the tests it was told to run, in the environment it had. Now every change that touches autoloading runs the suite with the same flag CI uses, before it’s pushed, and the review agent runs that command itself instead of trusting the implementer’s number.

The general rule: local verification has to match the environment that matters. If it can’t, you need to know exactly where it differs.

Verify the artifact, not the exit code

During an audit in September I found that Pressa had no database backups. None. The nightly job had reported success every night.

The production image didn’t include pg_dump. The job piped its output through gzip, so the exit status was gzip’s, and gzip happily compressed nothing. Every night it wrote a 20-byte file and logged “Created backup (0.0 MB)”. The job queue showed zero errors because nothing raised. And the backup directory wasn’t a mounted volume, so even a real backup would have disappeared on the next deploy.

Every layer said “fine”. The fix was to check the thing the job is supposed to produce: the job now uses pg_dump’s own exit status, opens the archive with pg_restore --list and fails unless it lists tables, and the admin dashboard shows “Last backup: X ago” in red when it’s older than 30 hours.

Read every hit, don’t just count them

On the client project, an agent was writing a ticket and needed the number of places a method was called. It ran grep -c, got 5, and wrote “five call sites” into the ticket. The fifth hit was the method’s own definition. To make the number fit, the ticket also described a fifth call, one that doesn’t exist anywhere in the code.

This is the failure mode that worries me most, because it’s plausible. Nobody double-checks the fifth item in a list of five. The rule now: any number that goes into a ticket or a report is read, not counted. You open every hit and look at it.

Check from different angles, and show which ones ran

When I ask an agent to analyze a request, one pass through the code is not an answer. But four greps going deeper into the same code are also one pass. What counts is the angle: the code, the documentation and tickets, the running system on staging, and the primary source, like the gem’s source code or the actual migration.

The rule came from a day when an analysis looked thorough after four rounds through the code. The documentation angle immediately showed that the field name it proposed was already in use with a different meaning. The staging angle disproved a claim the analysis had just made about how often a code path runs.

The rule was written down, and it was still skipped once, by a session that had read it at the start. So now every analysis ends with a line that lists the angles that actually ran:

Passes: code ✅ · docs + tickets ✅ · staging ✅ · primary source ➖ (not needed: no gem behavior involved)

A missing angle is visible at the moment of delivery, not three messages later. A rule in prose relies on good intentions. A line that has to be filled in doesn’t.

Look at production data

Tests check the cases you thought of. Production shows you the rest.

At the end of August, a free Pressa account ended the month at 51 compilations out of 50. Three requests had arrived within 30 milliseconds. Each one checked the count before any of the others had been recorded, and all three went through. No test had sent three requests at once. The fix was a per-account advisory lock in Postgres, with the count repeated inside it, and one rule: every accepted compilation is recorded through a single method that does exactly that, from every entry point.

It didn’t show up in a test or in a code review. It showed up in the usage numbers.

Check the environment the agent ran in

For a week, the agent on my home server implemented, reviewed and verified 16 client tickets on a smaller model than the one I intended. The command-line call didn’t pass a model flag, so it used the account default. The tickets were merged. The tests were green. Nothing in the output looked different.

Now the runner reads which model actually answered after every call and writes any mismatch into its log. The same thinking applies to rules and memory: if the agent didn’t load your project instructions, everything it does afterwards is wrong in ways the output won’t show. I wrote about those failures in A ticket goes in, a reviewed merge comes out.

Every review finding gets a decision

On the client project, pull requests are also reviewed by Codex. One day five of my pull requests went back on hold. On two of them, Codex findings had arrived four and six minutes after the last review request, and nobody saw them. On another, a finding had been dismissed with “Not in this PR”, although the new code relied on exactly that weakness.

So every finding now gets an explicit decision, written as a reply in that finding’s thread: either it’s fixed, or it’s rejected with evidence. Evidence means a test that disproves it, or a file:line on the base branch showing the same behavior already exists and the new code doesn’t depend on it. “Not in this PR” is not evidence. Before a pull request is handed over, a script checks that no finding is unanswered and that the last review verdict is for the current commit, not for the one before it.

It works in both directions. Codex once claimed that an ID larger than the database’s integer range would crash an endpoint. The agent checked it against the running app: Rails turns that lookup into WHERE 1=0 and returns an empty result, no exception. It posted the evidence on the pull request and stopped. The agent was right, and the empty result was exactly what the ticket asked for. Agents shouldn’t blindly obey a reviewer either.

Watch what real users run into

Tests check the code. They don’t check whether the person on the other end got what they came for.

In September, a user on Pressa’s free plan sent a 164 KB document three times. The free plan’s source limit is 30 KB, so each time the API answered with a correct, well-formed 413 error. Then the user left. Nothing was broken. Every check passed. The product still failed that person.

Now an account whose document is over its plan’s limit gets the first pages of the PDF and a clear note about which plan covers the whole document. And a script I run against production measures how many first calls succeed, and how many failures come back with a specific diagnosis instead of a raw TeX log. An unexplained failure that keeps repeating is the next thing to fix.

The short version

  • A fresh reviewer that sees only the diff, repeated until it finds nothing.
  • Run what CI runs, the way CI runs it.
  • Verify the artifact a job produces, not its exit code.
  • Read every hit behind a number, don’t just count them.
  • Check from different angles, and write down which ones ran.
  • Look at production data.
  • Check which model ran and which rules it loaded.
  • Give every review finding a decision with evidence.
  • Watch what real users run into, not only what the tests check.

None of this is specific to AI. It’s what good teams do anyway. The difference is volume: when an agent produces three times the code, the checking has to scale with it, or the failures do.