Tabbit01
Cleaner information
Page structure, interactive elements and state reach the model with less noise, so the agent spends fewer tokens looking for the target.
Browser agent benchmark · 25 tasks × 3 runs × 3 solutions
The fastest, leanest browser for AI agentsBeyond Codex, at state of the art
Tabbit is an AI browser built both for you and your agents. We compared Tabbit CLI with the Codex Chrome takeover and Vercel Labs Agent Browser in a detailed benchmark. The result: Tabbit CLI was the fastest and spent the fewest tokens in this test.
Install Tabbit, open it, then type /tabbit in any agent to try Tabbit CLI's browser-use setup.
Full suite · 75 runs per solution · blind-judgedAll 75 runs
TabbitBest
- Correct · 64.0%
- 48of 75 runs
- Median wall time
- 121.0s
- Input tokens per correct answer
- 803K
Codex Chrome
- Correct · 62.7%
- 47of 75 runs
- Median wall time
- 141.2s
- Input tokens per correct answer
- 1.57M
Agent Browser
- Correct · 60.0%
- 45of 75 runs
- Median wall time
- 209.6s
- Input tokens per correct answer
- 2.06M
Scroll for the full data
Same question set, same model, same machine
The comparison only means something if the three solutions faced the same test under the same conditions. Here is exactly what they faced.
Where the 25 questions came from
25 × 3 = 75The questions were taken from the BU Bench collection of 100 tasks, not written for this experiment. We sampled five tasks from each of its five sections; the specific tasks are not disclosed on this page to reduce benchmark gaming.
| Task type | Source | Tasks |
|---|---|---|
| Web reading | WebBench | 5 |
| Research | Mind2Web 2 | 5 |
| Web interaction | BU Bench custom | 5 |
| Fact QA | GAIA | 5 |
| Multi-hop retrieval | BrowseComp | 5 |
Run protocol
gpt-5.6-luna- All three solutions used the same model, gpt-5.6-luna, at medium thinking effort with a 30-minute cap, each in a brand-new model session.
- For every question the three solutions started at nearly the same moment; the next question began only after all three had finished. They shared one Mac and one network.
- The full 25-question suite ran three times, each round with a different pre-registered question order. No cherry-picking from the three rounds. That is 75 runs per solution and 225 runs in total.
- Tabbit ran on a fixed local instance with its own profile, launched with --use-mock-keychain and --no-proxy-server.
- Codex Chrome ran in a dedicated experiment tab that was closed after each run.
- Agent Browser ran as a later single-arm experiment using Vercel Labs agent-browser 0.35.2.
How answers were scored
- Each answer was handed to a fresh gpt-5.6-luna session at high thinking effort for independent scoring.
- The judge never saw which browser produced the answer, and never compared the three answers side by side.
- The judge saw the question, the final state, the final answer and the cited sources. The 10 GAIA and BrowseComp questions came with a ground-truth answer; the other 15 were checked against the task requirements.
- The judge had no browser tools: it did not open the cited pages, and it did not read full action logs, the DOM or screenshots. A self-reported "completed" was not counted as correct.
How the numbers are defined
Wall time
Real time from the agent process starting to it handing over a result. It covers model thinking, browser actions and CLI waits, but not the scoring that happened afterwards.
Median
The middle value once the runs are sorted. Less sensitive than the mean to a single timeout.
P90
90% of runs finished within this time. It shows how bad the slow tail is.
Tool calls
Every tool call the model made in that run — not just clicks and navigation.
Tokens
Input and output tokens recorded in run.json. They measure resource consumption, not a dollar price.
Cost per correct answer
Everything a solution consumed across all 75 attempts — time, tokens, tool calls — divided by the number of answers it got right.
Experiment process
From a fixed question list to a blind verdict, in four steps.
01Freeze the question listBefore any run started, five tasks were sampled from each of the five BU Bench sections and fixed for all three solutions.
02Start all three at onceFor each question the three solutions launched at nearly the same moment. The next question waited until all three had reported.
03Three pre-registered roundsThe whole suite ran three times, each round with a different order that was registered in advance. Results were not selected from the best round.
04Blind scoringAnonymous answers went to a fresh judge session that could not see the solution names and never compared answers across solutions.
Tabbit is the fastest and spends the least
Correctness is not the whole story when an agent runs at scale. What a solution costs per correct answer is what shows up on the bill and in the queue.
The real cost of one correct answer
All 75 runsTake everything a solution consumed across its 75 attempts — time, tokens, tool calls — and divide it by the number of answers it got right.
Time per correct answer
Tabbit219.3s
Codex Chrome227.9s
Agent Browser422.9s
Input tokens per correct answer
Tabbit803,144
Codex Chrome1,574,503
Agent Browser2,057,883
Output tokens per correct answer
Tabbit7,417
Codex Chrome7,686
Agent Browser10,266
Tool calls per correct answer
Tabbit20.1
Codex Chrome24.8
Agent Browser41.7
1.9×faster per correct answer than Agent Browser
−49%input tokens per correct answer, against Codex Chrome
−52%tool calls per correct answer, against Agent Browser
Tokens are not an abstraction — input tokens drive the bill and the context pressure of every call. All three solutions ran the same model, so the gap comes from what the browser hands the agent: cleaner information and shorter action paths.
Speed: not just faster on average
All 75 runsAll 75 runs per solution, then the same view restricted to runs that were scored correct.
| Solution | Correct / 75 | Median wall time | P90 wall time | Total wall time |
|---|---|---|---|---|
| Tabbit | 48 · 64.0% | 121.0s | 220.6s | 2.9h |
| Codex Chrome | 47 · 62.7% | 141.2s | 224.2s | 3.0h |
| Agent Browser | 45 · 60.0% | 209.6s | 489.0s | 5.3h |
Correct runs only
Same tasks, same rounds, only runs scored correct.| Solution | Correct / 75 | Median wall time | P90 wall time | Mean wall time |
|---|---|---|---|---|
| Tabbit | 48 · 64.0% | 118.3s | 225.6s | 137.0s |
| Codex Chrome | 47 · 62.7% | 131.6s | 226.7s | 140.0s |
| Agent Browser | 45 · 60.0% | 185.1s | 481.3s | 230.6s |
- Median wall time is 14% shorter than Codex Chrome and 42% shorter than Agent Browser.
- The slow tail is where it shows most: 10% of Agent Browser tasks ran past 489 seconds, while Tabbit’s P90 is 220.6 seconds.
- Across the whole suite, Tabbit finished in 2.92 hours of wall time; Agent Browser needed 5.29 hours.
- Looking only at correct runs, the average is 137.0s against 140.0s and 230.6s, and the median is 118.3s against 131.6s and 185.1s.
Tokens and tool calls
All 75 runsTotals across all 75 runs per solution, plus the average across correct runs.
| Metric | All 75 runs | Correct runs only |
|---|---|---|
| Input tokens | 38.55M / 74.00M / 92.60M | 550K / 954K / 1.16M |
| Output tokens | 356K / 361K / 462K | 5K / 5K / 6K |
| Tool calls | 965 / 1,164 / 1,875 | 13.6 / 15.4 / 24.3 |
- Tabbit’s total input tokens are 52% of Codex Chrome’s and 42% of Agent Browser’s.
- Total tool calls: 965 against 1,164 and 1,875 — 17% and 49% fewer.
- Average input tokens per correct run: 549,906 against 954,168 and 1,164,035.
Correctness
Every answer was scored by a judge that did not know which browser produced it, so "completed" never counted on its own.
- Correct answers out of 75: 48 (64.0%), 47 (62.7%) and 45 (60.0%). The gap is small — one run separates the first two.
- Questions answered correctly in at least two of three runs: 16, 14 and 15 out of 25.
- No solution dominated every task type. On multi-hop retrieval Codex Chrome scored 10 of 15 against Tabbit’s 9; on web reading Tabbit scored 11 against 9.
Correct answers by task type (15 runs each)
Agent Browser ran as a later single-arm experiment, so no like-for-like per-task-type figures exist for it. This table compares Tabbit and Codex Chrome only.| Task type | Tabbit | Codex Chrome |
|---|---|---|
| Web reading | 11 / 15 | 9 / 15 |
| Research | 8 / 15 | 8 / 15 |
| Web interaction | 15 / 15 | 15 / 15 |
| Fact QA | 5 / 15 | 5 / 15 |
| Multi-hop retrieval | 9 / 15 | 10 / 15 |
Where each solution stands
| Solution | Strengths | Main issues |
|---|---|---|
| Tabbit | Highest correctness on the full 25-question suite; lowest time, input tokens and tool calls per correct answer; shortest P90 tail. | Four run failures during the main experiment. |
| Codex Chrome | Best multi-hop retrieval; every web-interaction task succeeded. | Slowest and most tool-hungry on the runs all solutions got right; nearly twice Tabbit’s input tokens per correct answer. |
| Agent Browser | Middle of the pack on interaction and on questions answered correctly twice; open source and self-hostable. | Highest median time, highest P90, longest total wall time and the largest token consumption. |
Same model — the difference is what the browser tells the agent
All three solutions ran the same model at the same effort. So the gap is on the browser side of the loop.
02
Shorter paths
Common navigation, reading and form-filling actions collapse into fewer steps, so the model does not have to retry its way to the goal.
03
Fewer calls
965 tool calls across 75 runs — 17% fewer than Codex Chrome and 49% fewer than Agent Browser. Fewer calls means less context burned and a shorter tail.
Data source
The experiment data comes from the Tabbit Browser team. The detailed run records are not included on this page — to talk it through, email support@tabbit.ai.
Questions about the numbers
How much faster is Tabbit CLI than Codex Chrome?
Per correct answer, Tabbit averaged 219.3 seconds against 227.9 seconds, and finished the median run in 121.0 seconds against 141.2 seconds. The bigger gap is cost: 803,144 input tokens per correct answer against 1,574,503 — 49% fewer — and about 20% fewer tool calls.
How much faster is Tabbit CLI than Agent Browser?
Per correct answer, 219.3 seconds against 422.9 seconds, roughly 1.9× faster. It used 61% fewer input tokens and about half the tool calls. Median wall time was 121.0 seconds against 209.6 seconds.
Is the accuracy difference real?
Out of 75 runs Tabbit answered 48 correctly (64.0%), Codex Chrome 47 (62.7%) and Agent Browser 45 (60.0%). The difference is small — one run separates the top two — so it is not evidence that any one solution is reliably stronger.
How were answers judged correct?
Each answer went to a fresh judge session that never saw the solution name and never compared answers across solutions. It saw the question, final state, final answer and cited sources. A self-reported "completed" was not counted as correct. Ten questions had a ground-truth answer; the other 15 were checked against the task requirements.
Can these results be reproduced?
The question list, run settings, per-round summaries, every answer and every scoring result are preserved in the core data package. This page presents the summary, not a re-run.
What this benchmark can and cannot show
This is a practical comparison under one test setup, not a controlled scientific benchmark.
- The correctness results are close: 64.0%, 62.7% and 60.0%. This sample does not establish a reliable correctness advantage for any solution.
- The suite contains 25 tasks. More tasks, websites, task types and repeated batches would make the result more robust.
- Tasks ran serially while reusing browser environments. Ordering, residual session state, cache warming, throttling and long-run drift may have affected later runs.
- Browser profiles, user data, caches and the live state of websites were not completely identical. Agent Browser also ran later as a single-arm experiment, so this is not a strictly controlled scientific comparison.
Even with those limits, the results remain useful directional evidence: under this test setup, Tabbit consistently completed work faster and used substantially fewer input tokens and tool calls. The speed and resource-efficiency signal is larger than the small correctness gap, but it should still be validated on your own workloads.
Give your agent a browser that does not waste its context
Tabbit is an AI browser built both for you and your agents. In this benchmark, Tabbit CLI completed work fastest and used the fewest tokens against the Codex Chrome takeover and Vercel Labs Agent Browser.
Install Tabbit, open it, then type /tabbit in any agent to try Tabbit CLI's browser-use setup.