Gemini 3.1 Pro · Media / benchmark · Independent measurement
LayerLens Stratix covers 14,549 cases across six benchmarks and shows large task differences for Gemini 3.1 Pro between ARC and BIRD-CRITIC, among others.
Across 14,549 test cases on Stratix, LayerLens measured a wide task spread for Gemini 3.1 Pro Preview, from 92.3% on ARC-AGI-2 to 32.5% on BIRD-CRITIC. The results suggest that it is better suited to abstract reasoning, while SQL and repository-level engineering require routing or supporting tools.
Platform: LayerLens Stratix.
Sample: 14,549 test cases.
Categories/benchmarks: ARC AGI 2, LiveCodeBench, SWE Bench Lite, IF-Evals, BIRD-CRITIC, and BFCL v3, covering abstract reasoning, coding, software engineering, instruction following, SQL, and function calling.
Configuration: The article says it used standardized benchmark configurations and consistent parameters; it does not disclose each benchmark's complete prompt, temperature, number of repetitions, or confidence interval.
The article reports an evaluation of Gemini 3.1 Pro Preview dated 2026-02-19, but does not state whether it exactly matches the specific snapshot of the public API gemini-3.1-pro-preview. Testing was distributed across six standard benchmarks, and the results came from automated evaluations rather than selected demos.
| Benchmark | Score | Observation |
|---|---|---|
| ARC AGI 2 | 92.3% | Strongest result: abstract patterns and multi-step reasoning |
| IF-Evals | 78.4% | Generally good with multiple instructions, but fails on conflicting constraints |
| BFCL v3 | 71.2% | Function calling is above average, but complex multi-tool chains are unstable |
| LiveCodeBench | 61.0% | Standard coding tasks are acceptable; performance declines on long or multi-file tasks |
| SWE Bench Lite | 48.7% | Moderate performance on repository-level engineering; cross-file dependencies are difficult |
| BIRD-CRITIC | 32.5% | Weak at SQL and data reasoning; prone to errors with multi-table joins, subqueries, and schema inference |
This evaluation supports routing by task: try Gemini 3.1 Pro first for abstract reasoning and structured instructions; for SQL, repository-level code, and complex multi-tool chains, pair it with schema, test, and tool validation, or evaluate other models.
The 92.3% on ARC-AGI-2 does not match Google's published verified score of 77.1%. The difference may come from the dataset, harness, decoding, or model version; the rankings cannot be compared directly.
LayerLens did not publish the complete original inputs, run logs, seeds, costs, or per-question result downloads in the article. Reproduction requires Stratix or obtaining the data separately.
The article's author is LayerLens's marketing lead, and the platform has a product relationship; the evidence is stronger than an opinion but still requires third-party verification.
“2M context” does not match the 1,048,576 input tokens collected from the Google API model page; use the current specifications for the target endpoint.
Fix the model snapshot, API endpoint, thinking level, temperature, and max output.
Use the same versions of ARC AGI 2, IF-Evals, BFCL v3, LiveCodeBench, SWE Bench Lite, and BIRD-CRITIC; record each sample's input, output, tool calls, and failure reason.
Report all six scores and confidence intervals; do not let an average conceal the task gap between 92.3% and 32.5%.
Show the Google 77.1% baseline alongside these results and explain dataset/harness differences in the report.
The figures, task set, reasoning tier, and client conditions apply only to the listed source and collection snapshot. Different versions, harnesses, or providers must not be compared directly; undisclosed parameters remain unknown.
For a reproduction, fix the model version, provider or client, reasoning tier, tools, task-set version, sample count, and collection date, and record failures, retries, and human corrections. Full steps are in the source notes below.
LayerLens / Stratix · Jake Meany (LayerLens) · Original publication date 2026-02-19 · Site edit date 2026-09-20
Open original sourceGemini 3.1 Pro
Download the Tabbit client to check model access
