Across 657 tasks in Zapier AutomationBench 1.0.6, Gemini 4 Carbon's strict pass rates were 31.5% on Low, 53.0% on Medium, and 53.9% on High. The test measures whether end-to-end state in simulated business systems is fully correct, and is not equivalent to success rates on general office tasks.
Tasks this is useful for assessing: Comparing Carbon's pass rates, estimated cost per task, and tool-call statistics across its three reasoning tiers on this benchmark's business workflow tasks.
Tasks this should not be extrapolated to: These results cannot establish success rates for real Zapier automations, performance across all enterprise workflows, or actual billed costs. The test had no human clarification step and used simulated apps.
Model version: The report's model ID is gemini-4-carbon, and the test configuration is named Gemini 4 Carbon.
Test environment or client: API test on AutomationBench 1.0.6, with 47 simulated apps and about 500 API endpoints across six business areas: Sales, Marketing, Operations, Support, Finance, and HR. Each task allowed up to 50 steps.
Reasoning tiers and parameters: Low, Medium, and High. Other sampling parameters are not specified.
Each task starts with a small simulated company containing data in systems such as a CRM, email, spreadsheets, and support tickets. The Agent receives a business request and tools for searching, reading, updating, and sending. The data includes complications such as outdated records, similar names, and rules hidden in emails. The Agent cannot ask follow-up questions, and no human intervenes. At the end of each task, deterministic assertions check the final system state. A task passes only if every assertion passes. The model's written response itself is not scored.
Carbon was run from 2026-10-26 to 2026-10-27, with 657 tasks per tier. Zapier says leaderboard scores come from a held-out private evaluation set, while the public set of 600 tasks is for research and experimentation. The page does not disclose the number of tasks in the private set. The page reports typical run-to-run variation of less than 1%, but provides no confidence intervals.
| Tier | Strict pass rate | Passes / tasks | Cost per task | Total cost per passing task* |
|---|---|---|---|---|
| Low | 31.5% | 207 / 657 | $1.02 | $3.25 |
| Medium | 53.0% | 348 / 657 | $1.65 | $3.11 |
| High | 53.9% | 354 / 657 | $2.12 | $3.94 |
The estimated costs of all tasks, including failed tasks, divided by the number of passing tasks. All amounts are estimates in US dollars, not actual bills.
| Area | Low | Medium | High |
|---|---|---|---|
| Sales (117 tasks) | 38.5% (45) | 50.4% (59) | 56.4% (66) |
| Marketing (100 tasks) | 33.0% (33) | 64.0% (64) | 62.0% (62) |
| Operations (100 tasks) | 33.0% (33) | 65.0% (65) | 68.0% (68) |
| Support (100 tasks) | 32.0% (32) | 46.0% (46) | 46.0% (46) |
| Finance (120 tasks) | 35.8% (43) | 60.0% (72) | 60.0% (72) |
| HR (120 tasks) | 17.5% (21) | 35.0% (42) | 33.3% (40) |
Numbers in parentheses are counts of passing tasks. Each tier had 657 tasks in total. The area table shows that results do not rise monotonically with reasoning effort in every area. For example, Medium outperforms High in Marketing and HR.
Leaderboard version and snapshot: AutomationBench 1.0.6. The Zapier leaderboard lists Carbon High at 53.88% and $2.12 per task, and Medium at 52.97% and $1.65 per task. The report rounds these to 53.9% and 53.0% to one decimal place. Low is 31.5%, according to the Carbon-specific report.
Cost basis: Estimated using Google's list prices: $4 per million input tokens and $20 per million output tokens, with reasoning tokens billed as output. Cached reads are priced at $0.40 per million tokens (10% of the input price). Costs are estimates based on token usage for each tier's runs and do not represent actual billing.
Tool and step statistics: Average tool calls were 95.3 for Low, 89.0 for Medium, and 93.1 for High. Average step counts were 20.5, 16.4, and 16.6, respectively. Low had 39 tasks reach the 50-step limit, while Medium and High each had 3. Input cache shares were 90.1%, 86.8%, and 86.3%, respectively.
Leaderboard note: The page says the public set contains 600 research tasks and that the leaderboard uses a held-out private evaluation set. The report lists 657 tasks per tier but does not explain how that relates to the public set's task count. Use the per-tier denominators in the Carbon report and do not treat the two as the same dataset.
Within the same reported configuration and evaluation set, Medium is 21.5 percentage points above Low, while High is 0.9 points above Medium. High costs more than Medium, and Medium has the lowest estimated total cost per passing task.
The benchmark requires the final state to be fully correct. Partial completion does not count toward the official pass rate. The leaderboard also includes a partial_credit diagnostic value, but it is not part of the main score.
The results reflect task execution across 47 simulated apps. Permissions, interface reliability, data distributions, human review, and failure recovery may differ in real systems. The scores do not directly establish production performance.
The approximately 1% run-to-run variation is Zapier's summary. The report provides no confidence intervals or significance tests, so differences below that magnitude should not support strong conclusions.
Area results use a fixed task composition. The leaderboard summarizes area scores using each model's best reasoning tier, so its area rankings should not be treated as comparisons across a consistent tier.
The original pages are the Zapier AutomationBench leaderboard and the Carbon-specific results report. The report provides the version, run dates, model ID, tiers, task count, 50-step limit, main tool statistics, and cost assumptions. Reproduction still requires the same evaluation set version, Agent harness, full task configuration, model endpoint revision, and scoring assertions. The leaderboard is a snapshot that changes over time. This article records the 1.0.6 data displayed when it was collected on 2026-11-16.
Gemini 4 Carbon