Why do tests pass for the agent but fail later?

Tests pass for the agent but fail later for five common reasons: the agent ran them in a different environment, the passing result came from before its last edit, it ran only a narrow selection, something changed after its run, or the test is flaky. In each case both results were real; they just described different code, different tests or different conditions. The way through is to reproduce first and theorize second. Rerun the exact command the agent ran, at the exact revision it ran against, and compare with the run that failed. Where those runs differ, in command, revision, environment or result across repeated runs, tells you which cause you have. This page walks through that comparison, gives the fix for each cause, and shows how to bind a test result to the revision, command and environment that produced it, so a pass means the same thing next week. For results that go out of date in general, see stale test results.

Reproduce before you diagnose

Collect three facts about the agent's passing run from its transcript or log: the exact command, the revision or working tree it ran against, and where it ran. Then collect the same three for the failing run, usually CI or your own machine.

Check out the agent's revision cleanly and run the agent's command. If it passes, run the failing command on the same revision. If that fails, the difference is in the command: the test selection or its options. If both pass on your machine but CI fails, the difference is environment. If the agent's revision passes and the current one fails, something changed in between. If repeated runs of the same command on the same revision disagree, the test is flaky.

This sounds slow but usually takes minutes, and it replaces a long argument about whether the agent really ran the tests. A common surprise at this step is that the agent's revision cannot be identified at all, because it ran against uncommitted changes that were edited afterwards. That is itself the answer: the pass was stale before anyone looked at it.

Environment, stale results and narrow selection

Environment. Agents often run in a terminal with different versions, variables or services from CI: another runtime version, a cached dependency, a local database with leftover data, a different time zone or locale. Compare the environments directly and pin what differs, such as runtime and dependency versions.

Stale results. The agent ran the tests, made one more edit, and reported the earlier pass. Look at the order of events in the transcript. The fix is procedural: the check that counts is the one run after the final change, and it is rerun rather than quoted.

Narrow selection. The agent ran the tests for the area it changed and nothing else. That is sensible for speed and risky for shared code. In one of BRAIN FLUID's development tests, a shared money helper changed from rounding to truncation: the checkout test the change was made for passed, while a CSV export test in another module, which imported the same helper, failed with 123 instead of 124. Running only the checkout tests would have reported success. Narrow tests and missed regressions covers choosing a wider set.

Later changes and flaky tests

Later changes. The agent's pass was correct for its revision, and another change landed afterwards: a merged branch, a dependency update, a second agent's work. Bisecting between the agent's revision and the failing one finds the change that broke it, and the fix belongs to whoever owns that change, which may not be the agent. When the breakage crosses feature lines, when a fix breaks another feature goes through tracing it.

Flaky tests. The same command on the same revision gives different results. Common sources are timing and ordering, shared state between tests, real network calls, and anything that depends on the clock. Rerun the failing test alone several times; if it fails only sometimes, quarantine it and track it as its own task. Do not let an agent fix flakiness by retrying until it passes or by loosening assertions. Flaky tests and AI agents covers handling them without hiding real failures.

These two causes can look identical from a single failing CI run. Repetition separates them: later changes fail consistently, flaky tests do not.

Bind every result to what produced it

Most of this confusion disappears once a test result always travels with three things: the revision, the command and the environment.

Revision. Record the commit hash and whether the working tree was clean. A result from a dirty tree is evidence about code that may no longer exist.

Command. Record the exact command, including the test selection. 'Tests pass' is not a result; 'pytest tests/checkout passed at a1b2c3d' is, and it tells the reader at once that the export tests were not run.

Environment. Record at least the runtime version and where the run happened, such as the agent's terminal or CI.

With those three recorded, a later failure can be compared in minutes, and a result can be marked stale as soon as any file it covers changes. Make the full suite, or a selection chosen for the change's reach, the check required for done, and run it in the same environment CI uses. A pass then means the same thing to the agent, to you and to the next person who reads the record.

QUESTIONS

Why do tests pass locally but fail in CI after an AI agent's change?

Usually an environment difference: another runtime or dependency version, a missing variable or service, leftover local data, or a different time zone. Run the agent's exact command on its exact revision in a clean checkout, then in CI's environment. Pin whatever differs so the agent's runs and CI's runs test the same thing.

How do I check which tests an agent actually ran?

Read the command in its transcript or run log, not its summary. Look for a path, a name filter or a marker that limits the selection. If the agent ran only the tests next to its change, rerun with the full suite or with a selection that covers every caller of the code it touched.

Should an agent rerun tests after every edit?

It should rerun the required check after its final edit, at minimum, and report that run rather than an earlier one. Running a fast subset during the work is fine. The rule that matters is that the result counted toward done comes from the final state of the code, not an intermediate one.

RELATED GUIDES

• How to read BRAIN FLUID reports and provenance

RELATED HARNESS PAGES

• Stale test results Stale test results describe code that has since changed. Bind each result to a revision and rerun checks on every completion attempt.
• Why narrow acceptance tests miss AI regressions Why narrow acceptance tests miss AI regressions: a task's check can pass while a shared change breaks another consumer. A worked example and a fix.
• Flaky tests and AI coding agents Flaky tests and AI coding agents: how nondeterministic checks cause false completions and false failures, and the retry and quarantine rules that help.
• My AI agent fixed one bug and broke another feature Your AI agent fixed one bug and broke another feature? Find the shared code, find its other consumers, rerun their checks and record it as linked work.
• Self-reported vs independently run checks for AI agent work Self-reported vs independently run checks for AI agent work: why an agent's own "tests pass" is weaker than a check the system runs itself.