How do I know if a test is flaky or the agent's code is wrong?

Run the failing test many times against the unchanged base commit. If it fails there at any rate, it is flaky regardless of the agent's change. If it fails only with the change, run it repeatedly on the change too: consistent failure points at the code, while intermittent failure suggests the change made an existing race more likely.

Should AI agents be allowed to retry failing tests?

Within a fixed limit set in advance, and with every attempt recorded. One automatic rerun is a common rule. Without a limit, an agent can rerun until a flaky test passes and report success. A pass that needed retries should be reported as such, and a person decides whether it counts.

What does it mean to quarantine a flaky test?

The test keeps running but no longer blocks merges or task completion, while it sits on a tracked list with an owner and a task to fix it. Quarantine should be approved by a person, reported separately and reviewed regularly, or it becomes a quiet way to delete coverage.

Flaky tests and AI coding agents: how an AI project harness handles it

Flaky tests and AI coding agents: how nondeterministic checks cause false completions and false failures, and the retry and quarantine rules that help.

Flaky tests and AI coding agents are a bad combination because an agent treats every check result as a signal to act on. A flaky test, one that passes and fails on the same code, produces two kinds of error. A false completion: the flake happens to pass on broken work, or the agent reruns until it sees green and reports success. A false failure: the flake fails on correct work, and the agent "fixes" code that was fine, or edits the test. People learn which tests to distrust; an agent usually starts each session without that memory. The fix is a policy set before the run: a fixed retry limit with every attempt recorded, a fresh run on each completion attempt, and a quarantine list that only people maintain. This page explains the mechanisms and the rules. If an agent keeps repeating the same failing step, see an agent stuck in a loop; if it edits tests to get green, see test tampering.

Why flakiness hurts more with agents

A developer who sees a known-flaky test fail usually shrugs and reruns it. That judgment rests on memory: this test has failed on busy CI runners for a month. An agent session rarely has that memory. Each failure looks real, and each pass looks like proof.

That creates two loops. In the first, the agent sees a failure, changes code to address it, reruns, and the flake passes. The agent concludes its change fixed the problem and reports it. The diff now contains an unnecessary edit, sometimes a harmful one, such as an added sleep, a widened timeout or a retry wrapper around a real bug.

In the second, the agent's fix is actually wrong, but a flaky test sometimes passes anyway, and the agent reruns until a run comes up green. Rerunning until you see a pass is selection, not verification. If a broken change still passes a flaky test seven times in ten, three tries will almost always produce a green run. The report says "tests pass", which is literally true of the last run.

Both loops waste time and leave a record that looks like success. The second is the dangerous one, because it produces a self-reported pass on work that is broken.

Common causes worth checking first

Most flaky tests come from a short list of causes, and an agent can be pointed at them explicitly when a test is suspected.

Timing: fixed sleeps, timeouts close to real latency, assertions that race an asynchronous task.

Order dependence: a test that passes alone and fails after another test leaves state behind, or the reverse. Running the suite in random order exposes it.

Shared state: a database, a temporary directory, a module-level cache or an environment variable that is not reset between tests.

The outside world: network calls, real clocks, the current date, time zones, locale, and the order in which the file system lists a directory.

Randomness without a fixed seed, and floating point comparisons that depend on summation order.

Resource limits: tests that pass on a fast laptop and fail on a busy CI runner.

To confirm a suspicion, run the single test many times on unchanged code, then in random order with the rest of the suite. A test that fails at any rate on unchanged code is flaky. It should not be the deciding check for an agent's task until it is fixed or quarantined, because neither its passes nor its failures mean what they appear to mean.

Retry rules that do not hide anything

Retries are reasonable. Hidden retries are not. Set the rules before the run.

Fix the retry count in advance. One automatic rerun of a failed test is a common choice. The agent does not get to decide how many.

Record every attempt, not just the last. "Pass on attempt 3 of 3, failures on 1 and 2" is a flaky signal, not a pass. Treat any failure on unchanged code as information.

Rerun fresh on each completion attempt. A pass from an earlier attempt does not carry forward, because the code changed in between.

Cap total attempts per task. After the cap, the task stops and goes to a person instead of looping.

BRAIN FLUID keeps recorded attempts and outcomes with the work [C4], so a card that needed several attempts shows them. In development builds after v1.8.5, an owner-authorized project run has launch, retry, duration and failure limits [D4]; in those development builds the limits bound how often work is retried, but they do not tell a flaky check from a real failure, and no live AI model has run a multi-card project under that coordinator [D4].

Quarantine without hiding

When a test is confirmed flaky and cannot be fixed today, quarantine it properly.

Keep a quarantine list in the repository, owned by people. Each entry names the test, the date, the observed failure and a linked task to fix it. Agents may propose additions, but a person approves them. An agent that can quarantine tests can quarantine the one blocking its task.

Keep quarantined tests running and report them separately. They should not gate completion, but their results should stay visible, so you notice when one turns reliably red. That often means a real bug arrived under cover of the flake.

Replace the gate. If a quarantined test was the deciding check for a criterion, the task needs a different, deterministic check before an agent can complete it.

Report honestly. BRAIN FLUID's own validation reports show the pattern for broken tests: five legacy tests that fail at setup on a retired interface are listed as failures and not counted as coverage. A quarantined or broken test is a gap in coverage and should be reported as one, not dropped from the count.

Review the list monthly. Quarantine is a holding state, not a destination.

HOW BRAIN FLUID DOES IT

In the current v1.8.5 candidate, BRAIN FLUID runs work through verification gates and keeps recorded attempts, outcomes and reported usage with the work, so the attempts behind a result remain available to review ^[C4]. Evidence freshness shows whether recorded evidence still matches its supporting files ^[C6]. In development builds after v1.8.5, an owner-configured verification profile runs the required checks itself before a card can complete ^[D3]. In development builds, authorized project runs also carry launch, retry, duration and failure limits ^[D4]. In development builds, the gate runs only the configured checks; it does not detect flaky tests or keep a quarantine list ^[D3]. Those development build capabilities are not part of the v1.8.5 candidate and were tested with scripted agents or stand-in processes ^[D4].

C4 Harness gates and usage: Run work through verification gates and keep recorded attempts, outcomes and reported usage with the work. Evidence: Implemented; source and retained usage evidence
D4 Authorized project runs (development build): In development builds, an owner can authorize a bounded run over an explicit list of cards: one assignment at a time, with launch, retry, duration and failure limits, Pause, Resume and Stop, and the next eligible card selected only after a verified completion. Evidence: Development build (RC3 and RC4, October 30, 2026): deterministic three-card scenario with a scripted agent; the native CLI path was tested with a stand-in process. Not in the v1.8.5 candidate.
C6 Evidence freshness: See whether recorded evidence still matches its supporting files. Evidence: Implemented; source-documented and retained knowledge tests
D3 Independent completion checks (development build): In development builds, an owner-configured verification profile runs the required checks itself before a card can be completed; an agent cannot satisfy the gate with its own receipt. Evidence: Development build (autonomy phase 1 RC3, October 30, 2026): deterministic tests with a scripted agent and real check execution. Not in the v1.8.5 candidate.

QUESTIONS

How do I know if a test is flaky or the agent's code is wrong?: Run the failing test many times against the unchanged base commit. If it fails there at any rate, it is flaky regardless of the agent's change. If it fails only with the change, run it repeatedly on the change too: consistent failure points at the code, while intermittent failure suggests the change made an existing race more likely.
Should AI agents be allowed to retry failing tests?: Within a fixed limit set in advance, and with every attempt recorded. One automatic rerun is a common rule. Without a limit, an agent can rerun until a flaky test passes and report success. A pass that needed retries should be reported as such, and a person decides whether it counts.
What does it mean to quarantine a flaky test?: The test keeps running but no longer blocks merges or task completion, while it sits on a tracked list with an owner and a task to fix it. Quarantine should be approved by a person, reported separately and reviewed regularly, or it becomes a quiet way to delete coverage.

RELATED GUIDES

RELATED HARNESS PAGES

Why is my AI coding agent stuck in a loop? Agents loop when they lack real feedback, face contradictory constraints or chase flaky checks. How to spot a loop, set limits and decide to stop.
When an AI agent edits the tests to make them pass When an AI agent edits the tests to make them pass: the signs, why it happens, and how to keep test files out of an implementation task's scope.
Self-reported vs independently run checks for AI agent work Self-reported vs independently run checks for AI agent work: why an agent's own "tests pass" is weaker than a check the system runs itself.
What to do when an AI agent can't get the tests to pass When an agent can't get tests passing: stop the retries, separate environment from code failures, check the test, shrink the task, record the blocker.
Bounded autonomy: limits every unattended agent run needs Bounded autonomy: limits every unattended agent run needs, from an explicit task list to launch, retry, time, failure and spend limits.

BRAIN FLUID AI HARNESS

COMING SOON

Verified against the v1.8.5 development candidate (October 30, 2026 build) (reviewed October 30, 2026)

Free Windows software, not released yet. The download opens on the product page when the release is approved, with nothing to sign up for.

ABOUT BRAIN FLUID READ THE GUIDES GET INVOLVED