Across Zapier's 657 tasks, XHigh passed 13 more in total than High, raising the pass rate from 54.3% to 56.3% and the estimated cost per attempt from $0.88 to $1.69. Results by effort
What this evaluation can assess: Pass rates and costs across four reasoning tiers for cross-app business workflows that can be judged by their final state.
Limits on extrapolation: Workflows that require human clarification, success rates and bills for production automations, and latency in real-time interactions.
Model version: Google Gemini 4 Neon, listed in the report as gemini-4-neon.
Test environment or client: Zapier AutomationBench 1.0.6. The agent uses APIs to interact with simulated apps.
Reasoning tiers and parameters: Low, Medium, High, and XHigh, with a limit of 50 steps per task. The report leaves the exact model snapshot and other generation parameters unspecified. Methodology
In pre-release testing from 2026-11-17 to 2026-11-18, Zapier used the same 657 tasks for all four tiers. The tasks span sales, marketing, operations, customer support, finance, and human resources across 47 simulated apps. Each task starts with a preset company state. The agent receives a business request and works independently. Under task_completed_correctly, a task passes only if its final state satisfies every assertion. The report gives dates for the overall test run. Per-task runtimes and paired task-level results across tiers are unavailable. Methodology
Moving from Low to Medium yielded 24 more passes in total and raised the estimated cost per attempt by $0.18. Moving from Medium to High yielded 7 more passes and raised that cost by $0.30. The gains also vary by task type. In operations, Low, Medium, and XHigh each passed 68/100 tasks, while High passed 65/100. Results by effort · Results by domain
| Reasoning tier | Tasks | Passes | Strict pass rate | Estimated cost per attempt (USD) | Estimated spend per passing task (USD) | Share of input tokens cached |
|---|---|---|---|---|---|---|
| Low | 657 | 326 | 49.6% | 0.40 | 0.81 | 81.3% |
| Medium | 657 | 350 | 53.3% | 0.58 | 1.08 | 83.3% |
| High | 657 | 357 | 54.3% | 0.88 | 1.62 | 87.2% |
| XHigh | 657 | 370 | 56.3% | 1.69 | 3.01 | 88.5% |
Estimated spend per passing task = total estimated cost of all attempts ÷ number of passes. The numerator includes failed attempts. The table keeps the precision of the original report. Cost table · Cached input share
Zapier estimates costs from the token usage recorded in the tests. It uses list prices of $3 per million input tokens and $15 per million output tokens, counting reasoning tokens as output. When Zapier prepared the report, it assumed $0.15 per million tokens for cache reads. The report records no cache writes for Gemini's implicit cache. Usage, reasoning tier, and cache hits affect the estimates per attempt. Actual bills depend on measured usage and applicable rates. Cost methodology
Zapier's AutomationBench 1.0.6 measures the share of tasks that pass every assertion. Artificial Analysis's AutomationBench-AA is an independent implementation. Its primary metric is the share of task objectives completed without violating constraints. Check the benchmark and scoring method when comparing results. Zapier scoring method · AutomationBench-AA overview
The report's Results by effort, Cost, Tokens & tools, and Methodology sections give the task counts, pass counts, costs, cached input shares, and test configuration for verification. Zapier says it scores leaderboard entries on a held-out private task set and provides 600 public tasks for experimentation. When rerunning on the public set, use the same task set, starting state, tool configuration, and 50-step limit across tiers. Count a task as a pass only if every assertion passes, and record cache usage for each tier. Fully reproducing the leaderboard results requires the private task set and the original run configuration. Neon results report · Zapier leaderboard
Gemini 4 Neon