What to do when an AI agent can't get the tests to pass

When an AI agent cannot get the tests to pass, the usual causes are an environment problem that no code change will fix, a test that is wrong or out of date, or a task too large to debug in one piece. The first move is the same in all three cases: stop the session from trying again, because each extra attempt makes the code harder to read and the cause harder to see. Then read the failure yourself. An import error, a refused connection or a missing file is an environment failure. An assertion comparing two values is a code or test failure. That one distinction decides most of what follows. This page goes through the order: stop, classify, check the test, shrink the problem, and record the blocker so the next attempt does not start from zero. If the agent is cycling through the same edits rather than failing in new ways, start with why agents get stuck in loops instead.

Stop, then read the failure yourself

Interrupt the session once the agent has made two or three attempts that fail in the same place. Before anything else, save the state: commit or stash the working tree on a scratch branch so you can return to it, then look at the diff against where the task began. Agents under repeated failure tend to widen their edits, and you need to know how much has changed.

Now run the failing test yourself, alone, with full output. Agent sessions often see truncated output, or run the whole suite and read only the last lines. Running one test by name with verbose output often shows the cause at once.

Read the first error, not the last. A stack of failures usually has one root: a fixture that did not load, a module that failed to import, a setup step that raised. Everything after it is noise.

Finally, run the test on the base commit. If it fails there too, the agent did not break it; the task was started on a red baseline. That changes the conversation entirely, because the agent was being asked to fix something nobody had scoped.

Environment failure or code failure?

Sort the failure into one of two piles before trying to fix it.

Environment failures happen before your code's logic runs, or outside it: a module that cannot be imported, a missing environment variable, a database or service that is not running, a port already in use, a permissions error, a different runtime version, a network call in a test that cannot reach anything. Agents often try to fix these in code, adding fallbacks or mocks that hide the real problem. Fix them in the environment instead, and write the fix down so the next session starts from a working setup.

Code failures are assertions: expected 123, got 124. The test ran, the code ran, and they disagree. These are the ones worth handing back to the agent, with the exact output pasted in.

There is a third, smaller pile: failures that come and go on rerun with no change. Those are flaky, and retrying them only produces noise; flaky tests and AI agents covers how to isolate them. Do not let an agent chase a test that fails one run in five.

Is the test right?

An assertion failure means the code and the test disagree. It does not say which one is wrong. Check the test against the requirement, not against the new code.

Ask three questions. Does the test describe what the task asked for? If the task changed the rounding rule, a test asserting the old rule is now out of date, and updating it is part of the job. Was the test correct before this change? If it passed on the base commit and fails now, the change broke behavior someone relied on, and the default assumption should be that the code is wrong. Is the test testing something real? Tests that depend on the current date, dictionary ordering or the exact wording of an error message can fail for reasons unrelated to the change.

If the test needs to change, make that a decision a person takes, written down with the reason, not an edit the agent makes to reach green. An agent that edits the assertion until it passes has finished nothing; test tampering covers how to spot that afterwards.

Shrink the task and record the blocker

If the failure is a genuine code failure and the agent still cannot fix it, the task is usually too big to debug as one piece. Split it. Reproduce the failing case in the smallest test you can write, ideally one function call and one assertion, and give the agent only that. Once it passes, return to the original test.

If you stop for now, record the blocker so the next session does not repeat the same attempts. A useful blocker note has five parts: the failing test and the exact command; the first error, copied rather than summarized; what was tried and what each attempt changed; what has been ruled out, such as the environment or the test itself; and the current hypothesis. Mark the task as blocked, not in progress, so nobody assumes it is moving.

Deciding when an agent should stop and hand back, rather than keep trying, is a choice you can make before the session starts. When agents should stop and ask covers setting that threshold in the task itself.

QUESTIONS

Should I let the AI agent change a failing test?

Only after a person decides the test is wrong, with the reason written down. If the task changed the required behavior, updating the test is part of the work. If the test passed before the change, assume the code is wrong first. An agent editing assertions until they pass produces green output, not a fix.

How many times should an agent retry a failing test?

Two or three attempts that fail at the same point is a reasonable limit for an interactive session. After that, new attempts rarely add information and the diff keeps growing. Stop, read the first error yourself, classify it as an environment or code failure, and restart with a smaller task or a recorded blocker.

What if the tests were already failing before the agent started?

Then the task began on a red baseline and the agent cannot be judged against it. Run the suite on the base commit before assigning work and record which tests fail. Either fix those first as their own task or exclude them explicitly, so the agent and the reviewer agree on what passing means.

RELATED GUIDES

• How to read BRAIN FLUID reports and provenance

RELATED HARNESS PAGES

• Why is my AI coding agent stuck in a loop? Agents loop when they lack real feedback, face contradictory constraints or chase flaky checks. How to spot a loop, set limits and decide to stop.
• Flaky tests and AI coding agents Flaky tests and AI coding agents: how nondeterministic checks cause false completions and false failures, and the retry and quarantine rules that help.
• When an AI agent edits the tests to make them pass When an AI agent edits the tests to make them pass: the signs, why it happens, and how to keep test files out of an implementation task's scope.
• When an autonomous agent run should stop and ask When an autonomous agent run should stop and ask: failed checks, missing approval, open blockers, impact obligations, out-of-scope writes or uncertain state.
• Self-reported vs independently run checks for AI agent work Self-reported vs independently run checks for AI agent work: why an agent's own "tests pass" is weaker than a check the system runs itself.