Why narrow acceptance tests miss AI regressions

Why narrow acceptance tests miss AI regressions comes down to scope: a task's acceptance test checks the behavior the task asked for, while the change it accepts can reach every caller of the code it touched. When an agent edits a shared helper to satisfy one feature, that feature's test goes green and the breakage lands somewhere no task check looks. This is not specific to AI, but agents make it more frequent, because they reach for the shortest edit that satisfies the stated criterion and rarely survey the rest of the codebase unless asked. The fix is to choose an affected test set for each change, not only a task test. This page explains the mechanism, walks through a money-rounding probe from BRAIN FLUID's development testing in which exactly this happened, and shows how to find the tests that should have run. For the dependency-discovery side, see the blast radius of an AI change.

Task tests and change tests answer different questions

An acceptance test for a task answers: does the thing we asked for now work? A regression check answers a different question: did anything that used to work stop working? The first is defined by the task. The second is defined by the change, which is usually wider.

The gap is largest when the task's criterion and the change's reach differ. A task says "checkout must truncate fractional cents". The agent finds a shared cents() helper used by checkout and changes it, because that is the one-line fix. The task's test exercises checkout and passes. The helper also feeds the CSV export, the invoice PDF and a refund calculation. None of those were in the task, so none of their tests were selected.

Agents make this pattern common. They work from the task text, they favor the smallest edit that turns the named check green, and their session context rarely includes every importer of the function they changed. A developer who has worked in the codebase might remember that the export uses the same helper. The agent has no reason to look unless the task or the tooling tells it to.

Proportionate checks are still right for most tasks. The rule to add is that touching shared code widens the set.

A worked example: the money-rounding probe

On October 30, 2026, BRAIN FLUID's development build RC2 was tested with a disposable four-card project. ENG-1 added a shared cents() helper (DONE). ENG-2 required checkout to truncate fractional cents (TODO, depending on ENG-1). ENG-3, the CSV export, kept nearest-cent rounding (DONE); its import of the helper was deliberately left undeclared. ENG-4 delivered the app and depended on ENG-2 and ENG-3.

A scripted change turned round(value * 100) into int(value * 100) inside the helper. The checkout check total(1.239) == 123 passed. The export check row(1.235) == "124" failed, returning "123". That is the narrow-test failure exactly: the task's own check was green and another feature broke.

What the tooling did and did not do matters. The declared-relationship impact query, tested on that RC2 package, nominated ENG-2 and ENG-4 but not ENG-3, because the export's dependency was never recorded, and it correctly reported its coverage as incomplete [D6]. Evidence for the changed source flipped from current to stale. No regression card was created, ENG-3 was not reopened and nothing attributed the failure to the change. On that build's workflow route, a later report asserting a pass then completed the checkout card while the export test still failed. All edits and reports were scripted test stimuli; BRAIN FLUID did not write the change.

Choosing an affected test set

For any change to code with more than one caller, build the test set from the change outward.

Step 1: list the changed symbols. Functions, classes, constants, schema fields, configuration keys. git diff gives you the files; read it for the names.

Step 2: find their consumers. Use what your language offers: a repository search for the symbol name, your editor's find-references, an import graph tool or the build system's reverse dependencies. Look for indirect use too: string keys, reflection, templates and SQL that reference a column.

Step 3: map consumers to tests. The tests of each consuming module join the set. Where a consumer has no tests, that is a finding: add a characterization test that pins its current output before accepting the change.

Step 4: when in doubt, run everything. If the consumer list is long or uncertain, the full suite is cheaper than the investigation.

In the probe, step 2 with a plain text search would have found the export module's import of cents() in seconds, and its test would have failed before anyone marked the checkout task complete. Declared relationships are useful, but the source code is the authority on who imports what. More on that split in declared vs inferred dependencies.

Make shared code announce its consumers

The cheaper long-term fix is to make the affected set easy to find and hard to skip.

Give shared helpers contract tests that pin their behavior directly. cents(1.235) == 124 is a one-line test that would have failed on the truncation change regardless of which feature asked for it. Behavior changes to a shared helper then fail in the helper's own suite, and the task has to say explicitly that the contract is changing.

When one feature needs different behavior, prefer a new function over changing the shared one. ENG-2 wanted truncation; ENG-3 wanted rounding. Two functions, two names, no conflict. An agent will not usually make that design call unless asked, so put it in the task: "Do not change cents(); add a separate truncating helper if needed."

Record dependencies when you create work, and keep them current. A task whose code imports a shared module should say so, so that impact queries and reviewers can follow the link.

Finally, run the full suite before merge. Narrow checks gate the task; the full suite gates the branch. If a fix broke another feature anyway, that page covers tracing it back.

QUESTIONS

What is the difference between an acceptance test and a regression test?

An acceptance test checks that the requested behavior now works; the task defines it. A regression test checks that existing behavior still works; which ones matter depends on what the change touched. A change can pass every acceptance test for its task and still break a regression test elsewhere, especially when it edits shared code.

How do I find which tests to run after an AI agent changes a shared function?

List the changed symbols, then find every consumer with a repository search, your editor's find-references or an import graph. Run the tests of each consumer, add characterization tests where a consumer has none, and fall back to the full suite when the consumer list is long or uncertain.

Why did my AI agent change a shared helper instead of the feature code?

Because it was usually the shortest path to the stated check. The agent saw one function controlling the behavior and changed it, with no reason to consider other callers. Say in the task which shared code must not change, or ask the agent to list the callers of any shared function before editing it.

RELATED GUIDES

• How to read BRAIN FLUID reports and provenance
• How to read project progress in BRAIN FLUID Build

RELATED HARNESS PAGES

• The blast radius of AI code changes The blast radius of AI code changes: what one agent edit can break elsewhere, a worked rounding example, and how a reviewer finds affected features.
• Declared vs inferred dependencies in change-impact analysis Declared vs inferred dependencies in change-impact analysis: what task links and import graphs each catch, where each is blind, how to combine them.
• My AI agent fixed one bug and broke another feature Your AI agent fixed one bug and broke another feature? Find the shared code, find its other consumers, rerun their checks and record it as linked work.
• Choosing proportionate checks for each AI coding task Choosing proportionate checks for each AI coding task: one check that proves the criterion, a fast baseline, and clear rules for widening.
• Self-reported vs independently run checks for AI agent work Self-reported vs independently run checks for AI agent work: why an agent's own "tests pass" is weaker than a check the system runs itself.