CursorBench 3.2: multi-file agent success and cost

Historical snapshot notice

The body preserves research collected on September 8, 2026. The editorial summary and conditions above were reviewed on September 20. References to “current” values, rankings, and dynamic metrics in these notes describe the original snapshot, not live results. Reopen the source before making a decision.

One-sentence takeaway

On the current CursorBench 3.2 leaderboard, Gemini 3.8 Flash High scores 69.2% (9th among the 60 listed configurations), at an average of $2.38, 81,524 tokens, and 161 steps per task; Medium scores 67.0% (14th), at $1.93, 61,603 tokens, and 136 steps per task. Compared with Gemini 3.7 Flash on the same page, 3.8 is 7.6/8.0 percentage points higher on High/Medium, respectively, but consumes nearly twice the tokens and cost; this only shows higher scores and greater workload on Cursor's ambiguous multi-file Agent tasks, and cannot be extrapolated to general coding ability or API cost-effectiveness.

Applicable scenarios

• Tasks suitable for observation: Ambiguous, multi-file coding tasks from real Cursor sessions; codebase understanding, planning, fixing, refactoring, code review, and the instruction-following and advanced-tool-use tasks newly added in CursorBench 3.2.

• Tasks not suitable for direct inference: Single-file algorithm problems, programming in Chinese, general API calls, non-Cursor toolchains, long-running unattended Agents, production stability, or all software-engineering tasks.

• Model versions covered: The leaderboard explicitly lists Gemini 3.8 Flash High and Gemini 3.8 Flash Medium; this cannot be extended to undisclosed API snapshots, other providers, or Gemini 3.8 Flash Cyber.

• Applicable client, Agent, or API: The Cursor Agent's CursorBench 3.2 harness; this is not an end-to-end score for the Gemini API, Gemini App, or a self-built coding Agent.

• Recommended tier: Test High first for complex multi-file tasks; test Medium for tasks sensitive to budget and step count. The original page does not disclose enough of the temperature, system prompt, tool schema, retry, or context configuration to reproduce the result.

Test environment and method

• Task source: Cursor's methodology note says CursorBench uses Cursor Blame to trace committed code back to the Agent requests that generated it, creating pairs of developer queries and standard-answer solutions; many tasks come from internal codebases and controlled sources to reduce the risk of training contamination.

• Task format: Task descriptions are deliberately brief, underspecified, and potentially ambiguous, making them closer to real requests developers give Agents than to fully specified issue reports in public repositories. Compared with the initial version, CursorBench-3 roughly increased the lines of code and average file count, and covers more complex scenarios such as monorepo multi-workspace tasks, production-log investigations, and long-running experiments.

• Evaluation version: The current results page is CursorBench 3.2; the changelog says that instruction-following and advanced-tool-use problems were added on 2026-07-08. Version 3.1 added codebase-understanding, bug-finding, planning, and code-review tasks, and improved the scoring criteria for some editing tasks.

• Scoring dimensions: Cursor says it evaluates solution correctness, code quality, efficiency, and interaction behavior; the current public leaderboard reports only a composite SCORE, without Gemini 3.8 Flash's per-dimension scores, task count, or per-task scoring.

• Cost basis: The original page says average cost per task is calculated by multiplying the model's published input, cached-read, cached-write, and output prices by the tokens actually used for each task:

  Average cost per task = Σ (tokens of each type × corresponding price per million tokens)

• Online/offline boundary: The methodology article says Cursor also supplements offline CursorBench with controlled online analysis on real traffic to identify regressions where “the evaluator says it is correct but the developer experience is worse”; the current leaderboard does not publish Gemini 3.8 Flash's online metrics, so this article reports only the offline 3.2 results.

Results data

Current CursorBench 3.2 leaderboard

Among adjacent configurations on the same page, Gemini 3.8 Flash High's 69.2% is slightly below Fable 5.1 High's 69.4% and Opus 5 Extra High's 69.3%; Gemini 3.8 Flash Medium's 67.0% is slightly below GPT-5.6 Sol Max's 67.2% and above Opus 5 High's 66.7%. These configurations do not use a uniform model snapshot or the same reasoning tier, so small differences between adjacent entries should not be treated as stable wins or losses.

3.8 versus 3.7 on the same page

These are descriptive differences between leaderboard rows, not causal upgrade estimates under controlled variables. The Cursor page specifically warns that results have variance and that small score differences may not be statistically meaningful.

Conclusions

1. 3.8 is clearly higher than 3.7 on this Agent task set: High rose from 61.6% to 69.2%, and Medium from 59.0% to 67.0%; this direction is consistent with CursorBench 3.2's task design around multi-file work, tool use, and ambiguous requests.

2. The improvement comes with more execution: Gemini 3.8 Flash High averages 161 steps and 81,524 tokens, far above 3.7 High's 99 steps and 38,448 tokens; a higher score alone cannot be interpreted as “faster” or “cheaper.”

3. The extra High score has a cost: Compared with 3.8 Medium, High is 2.2 percentage points higher, but costs an additional $0.45 and uses 19,921 more tokens and 25 more steps per task. Because the page provides neither confidence intervals nor per-task results, this 2.2-point difference cannot be claimed to be statistically reliable.

4. This measures an Agent in the Cursor workflow: The task sources, Cursor tool environment, evaluator, and public-price cost basis jointly determine the result. Outside the Cursor harness, 69.2% cannot be directly treated as the raw coding accuracy of Gemini 3.8 Flash.

Limitations and boundaries

• Independence boundary of a platform-run benchmark: CursorBench was designed, run, and published by Cursor/Anysphere. Although it compares external models such as Google's, it is not a neutral-lab evaluation without a conflict of interest; the product's Agent harness and task distribution affect the results.

• Task set cannot be fully reproduced: Tasks come from real Cursor sessions, internal codebases, and controlled sources. The page does not publish the complete tasks, repositories, standard answers, evaluator, task count, number of runs, or per-task outputs.

• Incomplete configuration: The leaderboard publishes only High/Medium labels and aggregate scores, tokens, steps, and costs; it does not disclose the API snapshot, system prompt, temperature, context window, tool permissions, random seed, retries, or failure handling.

• Leaderboard position is not a model ranking: The current table treats multiple reasoning tiers of the same model as independent configuration rows; “9th” is a position among 60 configurations, not a ranking of 60 different models.

• Cost is not the bill: Cost is calculated from public token prices and the tokens actually used in the evaluation, which may differ from an account's cache hits, provider routing, discounts, and concurrent billing; steps are also not a universal definition of API request count.

• Statistical uncertainty: The original page explicitly says results have variance and that small score differences may not be statistically meaningful; it provides no confidence interval, single-run score, or failure sample for Gemini 3.8 Flash.

• Version and time: The 3.2 changelog entry is dated 2026-07-08, and this article records the current leaderboard opened on 2026-09-08; the page may later recalculate prices or refresh results, so this should not be mixed with other versions.

• Limited scope: Although the methodology note mentions controlled online analysis, the currently public table provides only offline leaderboard figures; it does not cover real-user satisfaction, long-duration autonomous-task success, Chinese-language contexts, or production-grade code security.

Reproduction recommendations

1. Fix the CursorBench 3.2 task-set version, repository commit, tool permissions, Agent system prompt, model snapshot, reasoning tier, context limit, stopping conditions, and retry strategy; if the internal task set cannot be obtained, clearly label the result an “independent CursorBench-style replication.”

2. Run Gemini 3.8 Flash High, Medium, and Gemini 3.7 Flash on the same task set; save each plan, tool call, edit diff, test result, token count, step count, wait time, and failure recovery.

3. Use multiple runs and report the mean, variance, or confidence interval; do not merely repeat 69.2% or the adjacent leaderboard positions.

4. Score solution correctness, code quality, efficiency, and interaction behavior separately, and record unauthorized edits, unverified claims, human takeovers, and rework time independently.

5. Calculate cost by input, cached read/write, and output tokens separately; also report the actual billing basis, and do not treat CursorBench's $2.38/task as the price of an arbitrary Gemini API request.

Original evidence and data

• Original CursorBench 3.2 leaderboard: https://cursor.com/cursorbench. The page defines the tasks as ambiguous, multi-file tasks from real Cursor sessions and lists Gemini 3.8 Flash High at 69.2%, $2.38, 81,524 tokens, and 161 steps, and Medium at 67.0%, $1.93, 61,603 tokens, and 136 steps; it also lists 3.7 High at 61.6% and Medium at 59.0%.

• The changelog on the same leaderboard shows that CursorBench 3.2 added instruction following and advanced tool use problems on 2026-07-08; the page also says average cost per task is calculated from public token prices and per-task token usage, and warns that small score differences may not be statistically meaningful.

• Cursor methodology note: https://cursor.com/blog/cursorbench. The article explains that tasks use Cursor Blame to trace Agent requests back from real commits, with many tasks from internal codebases and controlled sources; tasks remain underspecified, cover multi-file work, monorepos, production logs, and long-running experiments, and are supplemented by controlled online analysis alongside the offline benchmark.

Source excerpt or observation (short quote for compliance only)

The original CursorBench page summarizes the tasks as “ambiguous, multi-file tasks from real Cursor sessions,” while the methodology article emphasizes combining offline evaluation with online analysis on real traffic. This supports the core judgment that the result measures multi-file Agent performance in the Cursor workflow, not a general coding-ability score for Gemini 3.8 Flash.