Gemini 3.1 Pro · Media / benchmark · Independent measurement
MindStudio compares three flagships with HumanEval, SWE-bench, MATH, GPQA, MMLU Pro, and custom long-document tasks; Gemini's context advantage does not generalize to every code repair.
Unverified: the original source could not be rechecked. Historical figures below are not current verified results.
In a unified multi-model benchmark spanning code generation, long-form creative writing, graduate-level reasoning, mathematics, and long-document synthesis, Gemini 3.1 Pro holds an overwhelming advantage in its 2M-token ultra-long context and output cost (60% cheaper than Claude). However, it trails GPT-5.4 and Claude Opus 4.6 in complex multi-file bug fixing and literary-grade subjective writing.
Platform: MindStudio model evaluation and multi-agent orchestration pipeline.
Evaluated models: OpenAI GPT-5.4, Anthropic Claude Opus 4.6, Google Gemini 3.1 Pro.
Evaluated benchmarks and tasks:
Standard benchmarks: HumanEval (164 Python problems, pass@1), SWE-bench Verified (real-world GitHub issue resolution rate), MATH (competition-grade mathematics), GPQA Diamond (high-difficulty scientific reasoning), and MMLU Pro (comprehensive evaluation across 57 academic disciplines).
Custom empirical tasks: 5,000-word long-form literary writing, brand marketing copy (strict constraint following), 120k-token multi-document research synthesis report, and 6 categories of SVG spatial and visual code generation.
Evaluation method: Automated grading for objective coding and math tasks; blind review by 3 independent human evaluators for subjective tasks, scored across prose quality, instruction following, and narrative coherence.
All models used identical prompts and default temperature settings across standard benchmarks.
The long-document synthesis task used a uniform input of multiple research reports and academic materials totaling 80,000–150,000 tokens.
Pricing and parameter specifications:
GPT-5.4: Input $15.00 / 1M, Output $60.00 / 1M, Context 128k tokens, Generation speed ~80 TPS
Claude Opus 4.6: Input $20.00 / 1M, Output $100.00 / 1M, Context 200k tokens, Generation speed ~55 TPS
Gemini 3.1 Pro: Input $12.50 / 1M, Output $37.50 / 1M, Context 2M tokens, Generation speed ~75 TPS
| Benchmark | GPT-5.4 | Claude Opus 4.6 | Gemini 3.1 Pro | Analysis & Observations |
|---|---|---|---|---|
| HumanEval (pass@1) | 93.1% | 90.4% | 89.2% | Gemini tends to lock into incorrect assumptions prematurely on ambiguous prompts; performance is close on explicit algorithmic problems |
| SWE-bench Verified | 52.7% | 50.3% | 48.1% | On real-world cross-file repository repairs, Gemini partially closes the gap leveraging its large context window |
| GPQA Diamond | 83.9% | 87.4% | 82.1% | Claude leads in multi-step deep scientific derivations |
| MMLU Pro | 92.3% | 91.7% | 90.8% | All three achieve exceptionally high knowledge breadth (minimal gap) |
| MATH | 94.8% | 94.1% | 94.6% | All three are essentially on par in competition math (differences within error margins) |
| Test Task | GPT-5.4 | Claude Opus 4.6 | Gemini 3.1 Pro | Key Findings |
|---|---|---|---|---|
| 5,000-Word Literary Writing | 7.8 | 8.6 | 7.3 | Gemini satisfies plot requirements but produces relatively mechanical prose; Claude is best at pacing and subtext |
| Strictly Constrained Marketing Copy | 8.2 | 8.0 | 7.5 | GPT-5.4 adheres most strictly to negative constraints; Gemini's tone is more generic |
| 120k-Token Research Report Synthesis | Good (misses some deep subtle connections) | Excellent (best cross-document integration) | Solid (complete information retrieval, but summaries lean generic) | Gemini handles million-scale single-pass throughput effortlessly; Claude excels in nuance within 200k tokens |
| Complex SVG & Spatial Layout | Excellent (best layering and z-index) | Good (best at animations and flowcharts) | Fair (prone to adding redundant viewBox elements requiring manual cleanup) | Gemini exhibits slight shortcomings in complex spatial layout SVG code generation |
MindStudio provides clear guidelines for model selection:
Default first choice for coding and engineering execution: GPT-5.4 (highest accuracy, fastest generation speed).
High-quality long-form writing and extreme scientific reasoning: Claude Opus 4.6 (leads in prose quality and GPQA).
Ultra-long document retrieval, full codebase reading, and production-grade high-throughput cost-sensitive tasks: Gemini 3.1 Pro (2M-token context, output cost is only 37.5% of Claude's, offering the highest engineering cost-performance).
The empirical review was published in mid-March 2026; subsequent fine-tuning updates to each model may slightly alter scores.
Subjective scoring is constrained by the preference distribution of 3 evaluators; although blind review was employed, literary evaluation is inherently subjective.
Prepare the standard test environments for HumanEval and SWE-bench Verified.
Construct a 5,000-word creative writing brief and a 120k-token multi-document dataset, pinning the API endpoints for all three models.
Record pass@1 rates, operational costs, and actual throughput TPS, and organize three evaluator groups to conduct normalized blind scoring.
The figures, task set, reasoning tier, and client conditions apply only to the listed source and collection snapshot. Different versions, harnesses, or providers must not be compared directly; undisclosed parameters remain unknown.
For a reproduction, fix the model version, provider or client, reasoning tier, tools, task-set version, sample count, and collection date, and record failures, retries, and human corrections. Full steps are in the source notes below.
MindStudio Blog · Luis Chavez-Mattos (Director of Product, MindStudio) · Original publication date 2026-03-15 · Site edit date 2026-09-20
Open original sourceGemini 3.1 Pro
Download the Tabbit client to check model access
