Stale test results

Stale test results are passing (or failing) results that no longer describe the code in front of you, because the source, the tests, the dependencies or the configuration changed after the check ran. A result is a statement about one snapshot. Once anything in that snapshot moves, the result is history, not evidence. With AI coding agents this happens constantly: the agent runs the tests, then makes "one small cleanup", then reports the earlier green run. The fix is simple to state. Record which revision each result applies to, rerun the required checks on every completion attempt, and refuse to complete when the source changed after the check. This page explains what goes into a snapshot, how results go stale in agent work, and how to bind and enforce them with ordinary tools. It is about check results; for notes and decisions that drift out of date, see stale project memory, and for checks that pass and then fail in use, see tests that pass for the agent and fail later.

What a test result is actually about

A test result answers a narrow question: did this command, run against these inputs, exit successfully? The inputs are wider than most people track.

The source under test is the obvious one. The test files themselves are another: an edited assertion changes what "pass" means. Dependencies matter, through the lockfile and whatever is installed in the environment. Configuration matters: environment variables, feature flags, settings files, and the test runner's own config, which decides which files are collected. Data fixtures and generated files matter. Sometimes the platform matters too: the Python version, the time zone, the locale.

If any of those changes, the old result does not automatically become wrong, but it stops being evidence. It might still hold. You no longer know. The honest status of such a result is "stale", which is different from both pass and fail.

This is why a result without an identifier is weak. "Tests passed" with no commit hash, no note of uncommitted changes and no command line cannot be compared with anything. "pytest tests/payments exited 0 on commit 4f2a1c9, clean tree" can be checked against the code you are about to accept in a few seconds.

How results go stale in agent sessions

Agents produce stale results in predictable ways.

Edit after test. The agent runs the suite, sees green, then renames a variable, tidies imports or applies a review comment. The final report cites the earlier run. This is the most common case and the easiest to miss in a long transcript.

Rebase or merge after test. The branch was green against yesterday's main. Main moved, the branch was rebased, and nobody reran anything.

Another writer. A second agent or a teammate changed a shared file in the same working tree while the first agent's result sat in its report. Coordinating that is covered in stopping agents editing the same files.

Dependency drift. A lockfile update or a fresh install pulled in a new minor version of a library.

Partial reruns. The agent fixed one failure, reran only that test, then reported the whole suite as passing by combining two different runs in its summary.

In each case every individual statement in the report may be true. The run did pass. It just passed against code that no longer exists, and nothing in the report tells you so.

Bind each result to a revision

Binding means storing enough with each result to tell, later, whether it still applies.

At minimum, record the command, the commit hash, whether the working tree had uncommitted changes, the exit code and the time. If the tree was dirty, the commit hash alone is not enough: commit first, or record a hash of the changed files. For dependency-sensitive checks, include a hash of the lockfile. For checks that depend on configuration, note the relevant settings.

With that in place, staleness becomes a comparison instead of a judgment call. At acceptance time, compute the same identifiers for the code you are about to accept. If they match, the result applies. If anything differs, the result is stale and the check runs again.

One caution: a matching hash shows the bytes are unchanged, not that the check was any good. A trivial test bound perfectly to a revision is still a trivial test. Binding solves freshness, not adequacy, which is the subject of choosing proportionate checks.

BRAIN FLUID applies the same idea to recorded project evidence: it shows whether that evidence still matches its supporting files [C6].

Rerun on every completion attempt, refuse on change

The strongest rule is also the simplest: every attempt to complete a task reruns the required checks against the code being accepted, and completion is refused if the source changes between the check and the moment the result becomes final.

Rerunning costs time, so teams look for shortcuts. Caching is acceptable only when the cache key is the full snapshot: source, tests, dependencies and configuration. Reusing "the last green run on this branch" is not caching, it is hoping.

The second half of the rule closes a race that is easy to overlook. A check can pass at 10:02 and the agent can edit a file at 10:03, before the result is recorded as final. In development builds after v1.8.5, BRAIN FLUID's owner-configured verification runs the required checks itself before a card can complete [D3]. In a separate development build test, the source was changed after verification and before publication, and completion was refused. That is the behavior to aim for in any pipeline: the window between checked and accepted should be closed or detected.

In CI, the equivalent is requiring checks on the commit that will actually be merged, not only on an earlier branch head, and rerunning them when the base branch moves.

QUESTIONS

How do I know if a test result is still valid?

Compare the snapshot it ran against with the code you are accepting: commit hash, uncommitted changes, lockfile and relevant configuration. If all of them match, the result applies. If anything differs, treat the result as stale and rerun the check. A result recorded without a commit hash cannot be validated at all, so record one every time.

Should CI rerun tests after a rebase?

Yes. A rebase produces new commits whose combined code was never tested, even if the branch and the base were each green on their own. Require checks on the exact commit that will be merged, and configure the pipeline to rerun when the base branch changes. Skipping this is a common source of broken main branches.

Is it safe to cache test results to speed up checks?

Only if the cache key covers everything the result depends on: source, test files, dependencies, configuration and toolchain version. Build systems that hash their inputs do this well. Reusing the last green run on a branch, or a result reported by an agent, is not caching; it is assuming nothing changed.

RELATED GUIDES

• How to read project progress in BRAIN FLUID Build
• How to preview work with the BRAIN FLUID harness CLI

RELATED HARNESS PAGES

• Self-reported vs independently run checks for AI agent work Self-reported vs independently run checks for AI agent work: why an agent's own "tests pass" is weaker than a check the system runs itself.
• Stale AI project memory Stale AI project memory: how notes and evidence an agent reads fall out of date, how to detect it, and what a freshness check cannot tell you.
• Why do tests pass for the agent but fail later? Tests that passed for an agent can fail later because of the environment, stale results, narrow selection, later changes or flakiness. How to tell.
• evidence gates for AI work Evidence gates for AI work: separate proposed from approved execution, run checks, and record outcomes so acceptance rests on evidence.
• When a dependency changes, which approvals go stale? When a dependency changes, which approvals go stale? How evidence tied to source bytes goes stale, and what a stale flag does and does not tell you.