To compare browser agents, divide the cost of all attempts by the number of distinct tasks that passed an outcome check. Include failed runs and retries in the cost. Count a recovered task once. This gives you a budget unit you can compare across setups, even when one setup uses cheaper tokens but needs more attempts.
Define the result before measuring the cost
For a read-only job, require the requested fields to match the source. For a write, verify the destination record and its new value. A completed chat message is useful evidence about what the agent said. It does not confirm that a page loaded or that a record was saved. The first-task checklist and browser CLI guide describe checks you can adapt.
We ran Codex CLI 0.166.0 with the requested model label gpt-6.1-sol and Tabbit CLI 1.22.6.0 against a local read-only page. Success required an HTTP 200 tool observation plus the visible heading and both generated IDs in the tool output and the final answer. The second task deliberately returned HTTP 503 on its first attempt. We restored the page, then retried the same task with the same IDs.
| Task and attempt | Input tokens | Cached input | Output tokens | Browser result |
|---|---|---|---|---|
| A · 1 | 49,710 | 24,576 | 62 | 200, verified |
| B · 1 | 50,120 | 37,376 | 56 | 503, failed |
| B · 2 | 50,288 | 24,832 | 61 | 200, verified |
The three attempts took 13.56, 17.04, and 11.96 seconds wall time. These are durations for the attempts, not operator time. Each emitted one expected tool call. All client exit codes were zero. We had two verified tasks and three attempts. The controlled failure shows how to account for recovery. The sample does not estimate a production success rate.
Keep observed usage separate from prices
When cached tokens are a subset of input, calculate model cost as ((input − cached) × ordinary rate + cached × cache rate + output × output rate) ÷ 1,000,000, using rates per million. A provider may price additional categories separately. Use your invoice or pricing terms, and don't charge the same tokens twice. OpenAI prompt caching documentation shows why cache categories matter. The pricing guide and CLI benchmark are separate references, with their own models and workloads.
Here is a budget example applied to the observed counters. The rates are illustrative USD inputs. This is not a bill, a quote, or the price of the tested model.
| Budget line | Assumption | Cost for the three attempts |
|---|---|---|
| Model | $2 ordinary input, $0.20 cached input, $12 output per million | $0.1461728 |
| Browser and tool | $0.01 per attempt | $0.03 |
| Review | 30 seconds total at $30 per hour | $0.25 |
| Total per verified task | Sum divided by 2 | $0.2130864 |
Model only, the failed attempt contributed $0.0336352 under those assumptions. Dropping that line would hide the cost of reaching the verified result for task B. For a real budget, allocate recurring browser fees over the same cohort and record the actual human minutes. Distinguish time spent checking and recovering from automated elapsed time.
Compare complete cohorts
Use the same inputs, permission state, retry limits, and pass criteria for both setups. Report attempts, distinct tasks, verified completions, failure reasons, and total spend. With no completions, the ratio is undefined. For a quick illustration, $0.03 per attempt at 60% success costs about $0.05 per verified result. $0.04 per attempt at 90% costs about $0.0444, given one attempt per task and no other charges. Better outcomes can offset a higher attempt price.
For the next test, keep tool response sizes and screenshots visible in the logs. MCP and CLI can produce different input footprints even with the same task, and browser automation can separate simple deterministic checks from model reasoning. If your workflow relies on a signed-in page, include permission and session recovery in the test plan. Connect to an existing browser · Try the browser route in Tabbit
FAQ
How do I calculate the cost per successful task?
Add model, browser and tooling, and human review costs for all attempts in the same cohort. Divide by the number of distinct tasks that passed an outcome check.
Should failed attempts and retries count?
Yes. Add their actual usage to the numerator. A retry that succeeds contributes one task to the denominator, even if it took several attempts.
How should I handle cached tokens?
Confirm whether cached tokens are included in the input total. If they are, subtract them before multiplying by the ordinary input rate, then apply the cached rate separately.
What happens when no tasks succeed?
Report total spend, attempted task count, and zero verified completions. The cost per successful task is undefined in that cohort.
Does client exit code zero prove the task succeeded?
It confirms the client finished under its own rules. Verify the result in the browser or destination system. In our fixture one run exited zero while the page returned HTTP 503.