How to evaluate an AI coding workflow tool

How to evaluate an AI coding workflow tool, in one line: run the same real tasks with and without it, check every result with a test the agent did not write, keep every attempt including failures, and report what you could not measure. Demos are a poor guide because they show the best run of a task chosen to look good. What you need to know is how the tool behaves on your kind of work, across repeats, when things go wrong. That takes an afternoon of setup and a few hours of runs, not a research project. The method below works for any tool that sits around a coding agent: a harness, an editor integration, a prompt library, a context system or a new way of writing tasks. It also works for comparing two configurations of the same tool. Measuring usage is its own topic, covered in measuring token usage fairly; this page covers the evaluation as a whole.

Step 1: choose tasks with independent checks

Pick four to eight tasks that resemble your real work: a bug fix, a small feature, a refactor, a test-writing task, perhaps one that touches several files. Synthetic tasks are fine if they are honest copies of real patterns.

For each task, write a correctness check before any run. It must be independent of the agent: a test file the agent cannot see or must not edit, a script that compares output, or a precise manual checklist. Then run the check against the untouched starting code and confirm it fails. A check that passes before the work is done measures nothing.

Freeze each task as a fixture: a repository snapshot or a script that recreates it, plus the exact task text. Every run starts from a fresh copy, so one run cannot leave state for the next.

Watch for checks the agent can satisfy without solving the task, such as editing the test or special-casing the input. Test tampering by agents lists the patterns. Keep the check outside the workspace, or confirm after each run that it is unchanged.

Write down, before running, what result would change your mind. Otherwise any outcome will look like confirmation.

Step 2: run matched pairs, repeated

Each comparison is a matched pair: the same task, same fixture, same model, same effort or reasoning setting, same machine, one run in each condition. Change only the thing you are evaluating.

Repeat each pair at least twice. Agent runs vary, and a single pair can easily point the wrong way. More repeats are better; two is the minimum that shows you whether results are stable at all.

Drive the real tool the way you would use it, not a mock or a shortcut script, because the integration is part of what you are evaluating. If the tool has a user interface, run through it.

Decide your retry rule in advance and stick to it. No automatic retries. If a run fails for an infrastructure reason, record it and why, and replace it only with an explicit decision you write down. Never quietly rerun until the numbers look right.

As an example of scale, one benchmark during BRAIN FLUID development used 24 counted runs: six synthetic coding tasks, two conditions, two repeats, one small model at low effort, all driven through the actual Desk. Every task had an independent check that failed on the untouched fixture, and all 24 counted runs passed it. It is a small study, and it should be reported as one.

Step 3: record everything, including failures

For each run record: task, condition, repeat number, model and settings, start and end time, the correctness check result, whether the agent claimed success, files changed, and usage.

Keep failed and excluded runs in the data set, marked with the reason. An evaluation that drops failures will make any tool look reliable. In the benchmark mentioned above, three earlier attempts were excluded from the counted set but kept, including one failed run whose usage could not be recovered. That usage stayed unknown rather than being entered as zero, and its replacement was explicitly authorized.

Compare the agent's claim with the check. A tool that produces more confident success reports without more passing checks is making things worse, not better. This is the gap that self-reported vs independent checks describes.

Record usage as complete session ledgers, not a single number from the end of a run, and keep the categories separate. The measurement details are in the token usage article linked above.

Store the raw records, not just a summary table. Six months from now you will want to rerun one task with a new model, and you will need to know exactly what the old run did.

Step 4: test behavior under failure, then report limits

Pass rates and usage are half the picture. Also test how the tool behaves when things go wrong, because that is where workflow tools differ most.

Kill a run halfway and see what state is left and what the tool says about it. Give a task whose check cannot pass and see whether the tool records a failure or a success. Ask for something outside the declared scope and see what stops it. Close the tool mid-task and reopen it. Then look at the record and ask whether a colleague could reconstruct what happened from it alone. Run history and reports describes what that record should contain.

Then write the report with its limits stated plainly: how many tasks, which model, which effort level, synthetic or real, how many repeats, what was excluded and why, and what was not measured. A result from one model on six synthetic tasks says something about that setup and little about a large production codebase.

Be wary of any evaluation, including your own, that reports a single headline number without these details. The details are what let someone else decide whether the result applies to them.

QUESTIONS

How many test runs do I need to compare two AI coding setups?

At least two repeats of each matched pair, on four or more tasks, before you read anything into the results. Agent runs vary, so one run per condition can point the wrong way. If results flip between repeats, add repeats or tasks before concluding. Report the count either way, since small samples deserve small claims.

Why should a correctness check fail before the agent runs?

Because a check that already passes on the untouched code cannot tell you whether the agent did anything useful. Confirming the failure first shows the check is sensitive to the task. It also catches broken fixtures and checks that test the wrong thing, before you spend runs and attention on them.

Should I drop failed runs from an AI tool evaluation?

No. Keep them in the data with the reason they failed. Failures are part of how the tool behaves, and dropping them makes any tool look reliable. If a run failed for an infrastructure reason unrelated to the tool, you may replace it, but record that decision explicitly and keep the original run.

RELATED GUIDES

• How to read BRAIN FLUID reports and provenance

RELATED HARNESS PAGES

• Measuring token usage of AI coding agents fairly Measuring token usage of AI coding agents fairly: whole-session ledgers, separate categories, deduplicated records, failed runs kept and unknown never zero.
• Self-reported vs independently run checks for AI agent work Self-reported vs independently run checks for AI agent work: why an agent's own "tests pass" is weaker than a check the system runs itself.
• When an AI agent edits the tests to make them pass When an AI agent edits the tests to make them pass: the signs, why it happens, and how to keep test files out of an implementation task's scope.
• agent run history and reports Agent run history and reports: what to record about AI agent runs, how to read usage coverage honestly and where unknown differs from zero.