Measuring token usage of AI coding agents fairly

Measuring token usage of AI coding agents fairly means counting every token a whole session used, in separate categories, once each, including runs that failed, and marking anything you could not count as unknown. Many informal comparisons fall short on at least one of those points, and each one can flip a conclusion. A single total hides the fact that coding agents reread large amounts of cached context on every turn, so cache reads usually dwarf everything else. Per-message logs often repeat the same usage figures. Failed runs get quietly dropped. And a missing number becomes a zero that makes one condition look cheaper than it was. The method below addresses those problems with nothing more than the agent's own usage records and a spreadsheet. It is for comparing two setups on the same tasks, such as two context strategies or two prompt styles. It does not produce invoice figures, and it does not tell you what a session will cost. For the wider evaluation design, see how to evaluate an AI coding workflow tool.

Step 1: count whole sessions, not final messages

The unit of measurement is the whole agent session for one task, from launch to exit. An agent turn is not one model call: the agent reads files, calls tools, reads the results and calls the model again, often dozens of times. Usage printed for a single response, or estimated from the length of the final answer, misses almost all of it.

Collect usage from every model call in the session. Depending on the tool, that comes from its session transcript or log, a usage summary it prints, or the provider's usage records filtered to the session. If the agent spawns sub-agents or background calls, include them; they are part of what the task used. If your tooling already keeps run history with usage, start there, but check what it covers.

Fix the boundaries before you start. Does the session include the agent's initial project scan? The retries it did on its own? A summary step at the end? Any answer is fine as long as it is the same for every run in both conditions.

Write the ledger per run, not per condition: one row per model call with its identifiers and its usage fields. Totals come later. If you only keep totals, you cannot fix a counting mistake when you find one.

Step 2: keep the categories separate

Token usage for coding agents comes in at least four categories: fresh input, output, cache creation (input written to the prompt cache) and cache read (input served from it). Providers that support prompt caching report these separately and commonly bill them at different rates.

Keep all four as separate columns, per call and in every total, and report them separately. A single summed number is useful as a rough size, but it treats a cache-read token as equal to an output token, which neither the provider's accounting nor the model's work does.

Expect cache reads to be the largest category by far. Coding agents resend the conversation so far, including file contents and tool results, on every turn, and caching serves most of that repeated input. In a 24-run benchmark during BRAIN FLUID development, cache reads were by far the largest category in both conditions. This changes where to look: something that adds a little to every turn's context can matter more than something that shortens the final answer.

Do not convert tokens to money unless you have the provider's actual rates for every category and the run's actual model. Even then, the result is an estimate, not an invoice.

Step 3: deduplicate usage records

Raw agent logs can contain the same usage more than once. Some tools write one record per content block or per streaming update, and each carries the usage of the whole message it belongs to. Summing every record then counts a single model call two, three or more times.

Deduplicate by the identifier of the model response, so each call contributes its usage once. If the log has no stable identifier, find out how the tool writes records before you trust any total; a short test session with a known number of calls helps. Compare your deduplicated count of calls with what the session visibly did.

Watch the other direction too. Sub-agent calls, compaction or summarization steps, and calls made by hooks may live in separate files or logs. Missing them undercounts whichever condition triggers them more often.

Record how you deduplicated and keep the raw records alongside the ledger. A reviewer should be able to rerun your counting from the raw files and get the same numbers.

This step sounds pedantic. It is also the one most likely to produce a difference between conditions that is entirely an artifact of logging, especially when the two conditions produce different numbers of content blocks per response.

Step 4: keep failures and unknowns honest

Every attempted run belongs in the record, including ones that crashed, timed out or failed their correctness check. A failed run still used tokens, and a condition that fails more often is not efficient just because its successful runs were small. Report counted and excluded runs separately, and give the reason for every exclusion.

When usage for a run cannot be recovered, record it as unknown. Never enter zero. A zero silently pulls one condition's average down, while an unknown makes the reader account for the gap. Do not replace failed runs automatically; if a replacement is justified, authorize it explicitly and keep the original. The benchmark mentioned above excluded three earlier attempts but kept them, and one failed run's usage stayed unknown rather than counted as zero. Its replacement was explicitly authorized, with no automatic retries.

Compare usage only between runs that solved the task. Pair each usage figure with the result of an independent correctness check that failed on the untouched code, so a short run that did not finish cannot pass as an efficient one. For what to do when a tool reports nothing at all, see token usage shows as unknown.

QUESTIONS

Why are cache read tokens so high for coding agents?

Because a coding agent resends its growing conversation, including file contents and tool output, on every model call within a session. Prompt caching serves most of that repeated input from cache, so it is counted as cache reads rather than fresh input. Large cache-read totals are normal for long agent sessions and should be reported as their own category.

Should I add input, output and cache tokens into one total?

Report the categories separately first. A single total is useful as a rough size, but it weights a cache-read token the same as an output token, which neither billing nor model work does. If you publish a combined figure, publish the four categories next to it so readers can see what drives it.

How do I compare token usage between two prompts or context strategies?

Run both on the same tasks, model, settings and fixtures, with at least two repeats each. Collect whole-session usage per run, deduplicated, in separate categories. Keep failed runs and unknowns in the record. Compare only runs that passed an independent correctness check, and report the sample size and limits next to the numbers.

RELATED GUIDES

• How to read BRAIN FLUID reports and provenance

RELATED HARNESS PAGES

• How to evaluate an AI coding workflow tool How to evaluate an AI coding workflow tool without trusting demos: matched pairs, independent checks, every failure kept, and limits stated plainly.
• Why can't I see how many tokens my AI agent used? Token usage goes missing in interactive sessions, failed runs, uncounted cache categories and subagents. Why, and what to record instead of zero.
• agent run history and reports Agent run history and reports: what to record about AI agent runs, how to read usage coverage honestly and where unknown differs from zero.
• Can you put a hard spending cap on an AI coding agent? Client-side budget flags in agent CLIs are estimates that can overshoot. A hard cap needs a reservation before each request. Here are practical limits instead.