Self-reported vs independently run checks for AI agent work

Self-reported vs independently run checks for AI agent work is a question of who produced the evidence. A self-reported check is the agent telling you "all tests pass"; an independently run check is one that CI, a script you start or a harness gate executes against the current code and records without the agent in the loop. Only the second should be allowed to close work. The agent's report is still useful, because it tells you what the agent believes it did and where to look, but it comes from the same process that wrote the change, and that process can be wrong about the run, the scope or the code it ran against. This page compares the two, walks through a development-build test where an asserted pass completed a task while a real test still failed, and gives a pattern you can apply with any agent and any test runner. For the broader idea of gating work on evidence, see evidence gates for AI work; for the symptom side, see how to tell whether an agent really finished.

What a self-report can and cannot tell you

When an agent writes "ran the test suite, 42 passed", that sentence is a language model summarizing a terminal session. Usually it is accurate. The problem is that you cannot tell the accurate reports from the inaccurate ones by reading them, because both arrive with the same confidence.

The ways a self-report goes wrong are mundane. The agent ran a subset of tests (one file, a single test name) and summarized it as the suite. It ran the tests before its last edit and did not rerun them. It ran them in a different directory or virtual environment from the one your CI uses. It read a long log, missed a failure near the top and reported the summary line. It hit a timeout and described the run as passing because nothing had failed before the cut-off. Occasionally it states a result for a command it never ran.

None of this needs bad intent. The agent is grading its own work with its own reading of the output, under an instruction to finish. A self-report is a claim about evidence, not the evidence. Treat it as a pointer: it tells you which command to run and what result to expect, which makes your independent check faster to set up.

What makes a check independent

A check is independent when four things are true.

The definition belongs to someone other than the agent. The command, the test files and the expected result are set before work starts, and the agent cannot edit them as part of the same task. If the agent can rewrite the test, the check measures agreement with the agent, not correctness (see when an agent edits the tests).

The runner is not the agent. CI, a pre-merge job, a verification script you start yourself or a harness gate executes the command and captures its exit code directly. Nobody parses the agent's transcript for the word "passed".

The run happens against the code being accepted. The check runs at the completion attempt, on the exact revision, so an edit made after an earlier green run cannot ride on it. Stale test results are the usual way this breaks.

The outcome is recorded. Command, revision, exit code and time are stored with the task, so a later reader can see what was proven without trusting anyone's summary.

CI on a pull request meets most of these by default, which is why green CI is a stronger signal than "the agent said so". It still proves only what the checks cover.

A worked failure: the asserted pass that was accepted

BRAIN FLUID's own development testing produced a clear example. On October 30, 2026, a regression probe on development build RC2 used a disposable four-card project. A scripted change turned a shared money helper from round(value * 100) into int(value * 100). The checkout check total(1.239) == 123 passed. The CSV export check row(1.235) == "124" failed with "123". The app was broken.

On that build's local workflow route, a completion report carrying the failed check was refused with "Failed, missing or unsupported verification prevents completion." A later report asserting a pass, sent against the same still-broken app, was accepted, and the checkout card became DONE. The gate checked what the report said, not what the code did. In development builds on that route, a completion report is an agent assertion [D1].

The response was to stop trusting the report. In the later RC3 development build, owner-configured checks run independently before a card can complete [D3]. With int(value * 100) the real assertion failed with exit code 1, and the card stayed in VERIFY despite the agent's passing claim. With round(value * 100) the check passed and the card completed. All reports in these tests were scripted stimuli, not a live model's work. The lesson transfers to any setup: a gate that reads the agent's verdict is only as good as the verdict.

Making checks independent in your own workflow

Step 1: write the check into the task before the agent starts. "Done when pytest tests/test_checkout.py tests/test_export.py exits 0" is better than "make sure tests pass".

Step 2: keep the check out of the agent's edit area for that task. If the task is to change application code, test files and CI configuration are out of scope unless the task says otherwise.

Step 3: run it yourself, or let CI run it, against the final revision. With Claude Code or Codex CLI, let the session finish, then run the command in your own terminal or push the branch. Do not copy the result from the transcript.

Step 4: record the command, commit hash and exit code next to the task. A one-line note is enough.

Step 5: treat disagreement as information. If the agent said pass and your run fails, find out why before rerunning anything: wrong environment, stale run, partial run or a flaky test. Each points to a different fix.

This costs one extra command per task. It changes what "done" means from "the agent stopped and sounded confident" to "this check passed on this revision", which is the only version of done you can defend later.

QUESTIONS

Can I trust an AI agent when it says the tests pass?

Trust it as a lead, not as proof. The report is usually right, but you cannot tell from the text when it is wrong: the agent may have run a subset, run the tests before its last edit or misread a long log. Rerun the stated command yourself or in CI on the final commit, and accept the work on that result.

Is CI an independent check for AI-generated code?

Mostly, yes. CI runs a command the agent did not run, on a known commit, and records the exit code. Two gaps remain: if the agent edited the tests or the CI configuration in the same change, the check is no longer independent, and CI proves only what its tests cover. Review those files separately.

Why would a completion gate accept a failing task?

Because some gates read the agent's report instead of running the check. In a development-build test of BRAIN FLUID, a gate refused a report that listed a failed check, then accepted a later report that simply asserted a pass while the app still failed. A gate is only independent if it executes the check itself against the current code.

RELATED GUIDES

• How to preview work with the BRAIN FLUID harness CLI
• How to read BRAIN FLUID reports and provenance

RELATED HARNESS PAGES

• evidence gates for AI work Evidence gates for AI work: separate proposed from approved execution, run checks, and record outcomes so acceptance rests on evidence.
• How do I know whether an AI agent actually finished a task? Process exited, agent says done and verified are three different states. A checklist to tell which one your AI agent's task is really in.
• Stale test results Stale test results describe code that has since changed. Bind each result to a revision and rerun checks on every completion attempt.
• When an AI agent edits the tests to make them pass When an AI agent edits the tests to make them pass: the signs, why it happens, and how to keep test files out of an implementation task's scope.
• Why narrow acceptance tests miss AI regressions Why narrow acceptance tests miss AI regressions: a task's check can pass while a shared change breaks another consumer. A worked example and a fix.