In a unified multi-model benchmark spanning code generation, long-form creative writing, graduate-level reasoning, mathematics, and long-document synthesis, Gemini 3.1 Pro holds an overwhelming advantage in its 2M-token ultra-long context and output cost (60% cheaper than Claude). However, it trails GPT-5.4 and Claude Opus 4.6 in complex multi-file bug fixing and literary-grade subjective writing.
Platform: MindStudio model evaluation and multi-agent orchestration pipeline.
Evaluated models: OpenAI GPT-5.4, Anthropic Claude Opus 4.6, Google Gemini 3.1 Pro.
Evaluated benchmarks and tasks:
Standard benchmarks: HumanEval (164 Python problems, pass@1), SWE-bench Verified (real-world GitHub issue resolution rate), MATH (competition-grade mathematics), GPQA Diamond (high-difficulty scientific reasoning), and MMLU Pro (comprehensive evaluation across 57 academic disciplines).
Custom empirical tasks: 5,000-word long-form literary writing, brand marketing copy (strict constraint following), 120k-token multi-document research synthesis report, and 6 categories of SVG spatial and visual code generation.
Evaluation method: Automated grading for objective coding and math tasks; blind review by 3 independent human evaluators for subjective tasks, scored across prose quality, instruction following, and narrative coherence.
All models used identical prompts and default temperature settings across standard benchmarks.
The long-document synthesis task used a uniform input of multiple research reports and academic materials totaling 80,000–150,000 tokens.
Pricing and parameter specifications:
GPT-5.4: Input $15.00 / 1M, Output $60.00 / 1M, Context 128k tokens, Generation speed ~80 TPS
Claude Opus 4.6: Input $20.00 / 1M, Output $100.00 / 1M, Context 200k tokens, Generation speed ~55 TPS
Gemini 3.1 Pro: Input $12.50 / 1M, Output $37.50 / 1M, Context 2M tokens, Generation speed ~75 TPS
| Benchmark | GPT-5.4 | Claude Opus 4.6 | Gemini 3.1 Pro | Analysis & Observations |
|---|---|---|---|---|
| HumanEval (pass@1) | 93.1% | 90.4% | 89.2% | Gemini tends to lock into incorrect assumptions prematurely on ambiguous prompts; performance is close on explicit algorithmic problems |
| SWE-bench Verified | 52.7% | 50.3% | 48.1% | On real-world cross-file repository repairs, Gemini partially closes the gap leveraging its large context window |
| GPQA Diamond | 83.9% | 87.4% | 82.1% | Claude leads in multi-step deep scientific derivations |
| MMLU Pro | 92.3% | 91.7% | 90.8% | All three achieve exceptionally high knowledge breadth (minimal gap) |
| MATH | 94.8% | 94.1% | 94.6% | All three are essentially on par in competition math (differences within error margins) |
| Test Task | GPT-5.4 | Claude Opus 4.6 | Gemini 3.1 Pro | Key Findings |
|---|---|---|---|---|
| 5,000-Word Literary Writing | 7.8 | 8.6 | 7.3 | Gemini satisfies plot requirements but produces relatively mechanical prose; Claude is best at pacing and subtext |
| Strictly Constrained Marketing Copy | 8.2 | 8.0 | 7.5 | GPT-5.4 adheres most strictly to negative constraints; Gemini's tone is more generic |
| 120k-Token Research Report Synthesis | Good (misses some deep subtle connections) | Excellent (best cross-document integration) | Solid (complete information retrieval, but summaries lean generic) | Gemini handles million-scale single-pass throughput effortlessly; Claude excels in nuance within 200k tokens |
| Complex SVG & Spatial Layout | Excellent (best layering and z-index) | Good (best at animations and flowcharts) | Fair (prone to adding redundant viewBox elements requiring manual cleanup) | Gemini exhibits slight shortcomings in complex spatial layout SVG code generation |
MindStudio provides clear guidelines for model selection:
Default first choice for coding and engineering execution: GPT-5.4 (highest accuracy, fastest generation speed).
High-quality long-form writing and extreme scientific reasoning: Claude Opus 4.6 (leads in prose quality and GPQA).
Ultra-long document retrieval, full codebase reading, and production-grade high-throughput cost-sensitive tasks: Gemini 3.1 Pro (2M-token context, output cost is only 37.5% of Claude's, offering the highest engineering cost-performance).
The empirical review was published in mid-March 2026; subsequent fine-tuning updates to each model may slightly alter scores.
Subjective scoring is constrained by the preference distribution of 3 evaluators; although blind review was employed, literary evaluation is inherently subjective.
Prepare the standard test environments for HumanEval and SWE-bench Verified.
Construct a 5,000-word creative writing brief and a 120k-token multi-document dataset, pinning the API endpoints for all three models.
Record pass@1 rates, operational costs, and actual throughput TPS, and organize three evaluator groups to conduct normalized blind scoring.
Gemini 3.1 Pro
