Tabbit
ResourcesBlogModels
Tabbit LogoTabbit

Tabbit — The AI Browser that Works for You

Topics

  • Best Browsers 2026: Rankings and Top Lists
  • AI Browser Guides and Resources
  • Browser Comparisons: Head-to-Head Guides
  • Browser Reviews and Explainers
  • Browser Alternatives
  • Browser Directory: Guides by Browser
  • Agentic Browser Resources
  • Browser Downloads and Install Guides
  • Browser Productivity Resources
  • SillyTavern Model Guides and Presets

Popular Guides

  • Best Browsers 2026: Top 10 Ranked & Compared
  • Best Free AI Browsers 2026: Plans & Prices Compared
  • Rabbit Browser: What It Is & Why Tabbit Is Better
  • Firefox vs Chrome 2026: Privacy, RAM & Speed Compared
  • Best Lightweight Browsers 2026: 7 Tested for RAM & Speed
  • AI Browser List 2026: 10 Browsers Compared
  • Neobrowser: Norton Neo, Download & Alternatives
  • Edge vs Chrome 2026: Speed, RAM & Battery Tested
  • Best Internet Browser for Android 2026
  • Best AI Browser 2026: Top 9 Tested & Ranked

Events

  • Tabbit Skill Competition
  • KPOP SBTI Fandom Personality Test
  • Tabbit Campus Creator Program
  • fifi's Picks: AI Skills for Research Papers
  • User Survey

About

  • Tabbit Blog
  • Press & Media
English
简体中文English
Review
MediaGemini 4 Carbon

Gemini 4 Carbon on Zapier AutomationBench: Business Workflow Results

Original source

Zapier

AuthorNot specified

Tabbit curation2026-11-16

Read original

One-sentence takeaway

Across 657 tasks in Zapier AutomationBench 1.0.6, Gemini 4 Carbon's strict pass rates were 31.5% on Low, 53.0% on Medium, and 53.9% on High. The test measures whether end-to-end state in simulated business systems is fully correct, and is not equivalent to success rates on general office tasks.

Use cases

  • Tasks this is useful for assessing: Comparing Carbon's pass rates, estimated cost per task, and tool-call statistics across its three reasoning tiers on this benchmark's business workflow tasks.

  • Tasks this should not be extrapolated to: These results cannot establish success rates for real Zapier automations, performance across all enterprise workflows, or actual billed costs. The test had no human clarification step and used simulated apps.

  • Model version: The report's model ID is gemini-4-carbon, and the test configuration is named Gemini 4 Carbon.

  • Test environment or client: API test on AutomationBench 1.0.6, with 47 simulated apps and about 500 API endpoints across six business areas: Sales, Marketing, Operations, Support, Finance, and HR. Each task allowed up to 50 steps.

  • Reasoning tiers and parameters: Low, Medium, and High. Other sampling parameters are not specified.

Evaluation method

Each task starts with a small simulated company containing data in systems such as a CRM, email, spreadsheets, and support tickets. The Agent receives a business request and tools for searching, reading, updating, and sending. The data includes complications such as outdated records, similar names, and rules hidden in emails. The Agent cannot ask follow-up questions, and no human intervenes. At the end of each task, deterministic assertions check the final system state. A task passes only if every assertion passes. The model's written response itself is not scored.

Carbon was run from 2026-10-26 to 2026-10-27, with 657 tasks per tier. Zapier says leaderboard scores come from a held-out private evaluation set, while the public set of 600 tasks is for research and experimentation. The page does not disclose the number of tasks in the private set. The page reports typical run-to-run variation of less than 1%, but provides no confidence intervals.

Key results

By reasoning tier

TierStrict pass ratePasses / tasksCost per taskTotal cost per passing task*
Low31.5%207 / 657$1.02$3.25
Medium53.0%348 / 657$1.65$3.11
High53.9%354 / 657$2.12$3.94
  • The estimated costs of all tasks, including failed tasks, divided by the number of passing tasks. All amounts are estimates in US dollars, not actual bills.

Pass rates by business area

AreaLowMediumHigh
Sales (117 tasks)38.5% (45)50.4% (59)56.4% (66)
Marketing (100 tasks)33.0% (33)64.0% (64)62.0% (62)
Operations (100 tasks)33.0% (33)65.0% (65)68.0% (68)
Support (100 tasks)32.0% (32)46.0% (46)46.0% (46)
Finance (120 tasks)35.8% (43)60.0% (72)60.0% (72)
HR (120 tasks)17.5% (21)35.0% (42)33.3% (40)

Numbers in parentheses are counts of passing tasks. Each tier had 657 tasks in total. The area table shows that results do not rise monotonically with reasoning effort in every area. For example, Medium outperforms High in Marketing and HR.

Raw data

  • Leaderboard version and snapshot: AutomationBench 1.0.6. The Zapier leaderboard lists Carbon High at 53.88% and $2.12 per task, and Medium at 52.97% and $1.65 per task. The report rounds these to 53.9% and 53.0% to one decimal place. Low is 31.5%, according to the Carbon-specific report.

  • Cost basis: Estimated using Google's list prices: $4 per million input tokens and $20 per million output tokens, with reasoning tokens billed as output. Cached reads are priced at $0.40 per million tokens (10% of the input price). Costs are estimates based on token usage for each tier's runs and do not represent actual billing.

  • Tool and step statistics: Average tool calls were 95.3 for Low, 89.0 for Medium, and 93.1 for High. Average step counts were 20.5, 16.4, and 16.6, respectively. Low had 39 tasks reach the 50-step limit, while Medium and High each had 3. Input cache shares were 90.1%, 86.8%, and 86.3%, respectively.

  • Leaderboard note: The page says the public set contains 600 research tasks and that the leaderboard uses a held-out private evaluation set. The report lists 657 tasks per tier but does not explain how that relates to the public set's task count. Use the per-tier denominators in the Carbon report and do not treat the two as the same dataset.

Conclusions and limitations

  • Within the same reported configuration and evaluation set, Medium is 21.5 percentage points above Low, while High is 0.9 points above Medium. High costs more than Medium, and Medium has the lowest estimated total cost per passing task.

  • The benchmark requires the final state to be fully correct. Partial completion does not count toward the official pass rate. The leaderboard also includes a partial_credit diagnostic value, but it is not part of the main score.

  • The results reflect task execution across 47 simulated apps. Permissions, interface reliability, data distributions, human review, and failure recovery may differ in real systems. The scores do not directly establish production performance.

  • The approximately 1% run-to-run variation is Zapier's summary. The report provides no confidence intervals or significance tests, so differences below that magnitude should not support strong conclusions.

  • Area results use a fixed task composition. The leaderboard summarizes area scores using each model's best reasoning tier, so its area rankings should not be treated as comparisons across a consistent tier.

Reproduction notes

The original pages are the Zapier AutomationBench leaderboard and the Carbon-specific results report. The report provides the version, run dates, model ID, tiers, task count, 50-step limit, main tool statistics, and cost assumptions. Reproduction still requires the same evaluation set version, Agent harness, full task configuration, model endpoint revision, and scoring assertions. The leaderboard is a snapshot that changes over time. This article records the 1.0.6 data displayed when it was collected on 2026-11-16.

Curated by Tabbit

This is a third-party source navigator. Model versions, test environments, and personal experience vary; consult the original source.

Gemini 4 Carbon

Use and compare models in Tabbit

Gemini 4 Carbon

Related reviews

OfficialGoogle DeepMind

Gemini 4 Carbon Model Card: Knowledge Work and Multimodal Benchmarks and Their Limits

MediaArtificial Analysis

Gemini 4 Carbon: Artificial Analysis Reasoning Tiers and Index Costs

MediaAI IQ

Gemini 4 Carbon's Composite Score and Benchmark Coverage on AI IQ

MediaCursor's official CursorBench leaderboard2026-09-10

CursorBench 4.0: Gemini 4 Carbon on Multi-File Engineering Tasks

Gemini 4 Carbon

Related showcases & prompts

OfficialGoogle AI Studio / Gemini API

Google AI API Specifications for Gemini 4 Carbon

OfficialGoogle AI Studio / Gemini API

Gemini 4 Carbon Long Outputs: Continuation and Stop Conditions

OfficialGoogle AI Studio2026-11-10

Google AI Studio: Prompting Gemini 4 Carbon for Complex Tasks

OfficialGoogle AI Studio / Gemini API

Google AI Studio: Gemini 4 Carbon Agentic Video Understanding Configuration