Across 14,549 test cases on Stratix, LayerLens measured a wide task spread for Gemini 3.1 Pro Preview, from 92.3% on ARC-AGI-2 to 32.5% on BIRD-CRITIC. The results suggest that it is better suited to abstract reasoning, while SQL and repository-level engineering require routing or supporting tools.
Platform: LayerLens Stratix.
Sample: 14,549 test cases.
Categories/benchmarks: ARC AGI 2, LiveCodeBench, SWE Bench Lite, IF-Evals, BIRD-CRITIC, and BFCL v3, covering abstract reasoning, coding, software engineering, instruction following, SQL, and function calling.
Configuration: The article says it used standardized benchmark configurations and consistent parameters; it does not disclose each benchmark's complete prompt, temperature, number of repetitions, or confidence interval.
The article reports an evaluation of Gemini 3.1 Pro Preview dated 2026-02-19, but does not state whether it exactly matches the specific snapshot of the public API gemini-3.1-pro-preview. Testing was distributed across six standard benchmarks, and the results came from automated evaluations rather than selected demos.
| Benchmark | Score | Observation |
|---|---|---|
| ARC AGI 2 | 92.3% | Strongest result: abstract patterns and multi-step reasoning |
| IF-Evals | 78.4% | Generally good with multiple instructions, but fails on conflicting constraints |
| BFCL v3 | 71.2% | Function calling is above average, but complex multi-tool chains are unstable |
| LiveCodeBench | 61.0% | Standard coding tasks are acceptable; performance declines on long or multi-file tasks |
| SWE Bench Lite | 48.7% | Moderate performance on repository-level engineering; cross-file dependencies are difficult |
| BIRD-CRITIC | 32.5% | Weak at SQL and data reasoning; prone to errors with multi-table joins, subqueries, and schema inference |
This evaluation supports routing by task: try Gemini 3.1 Pro first for abstract reasoning and structured instructions; for SQL, repository-level code, and complex multi-tool chains, pair it with schema, test, and tool validation, or evaluate other models.
The 92.3% on ARC-AGI-2 does not match Google's published verified score of 77.1%. The difference may come from the dataset, harness, decoding, or model version; the rankings cannot be compared directly.
LayerLens did not publish the complete original inputs, run logs, seeds, costs, or per-question result downloads in the article. Reproduction requires Stratix or obtaining the data separately.
The article's author is LayerLens's marketing lead, and the platform has a product relationship; the evidence is stronger than an opinion but still requires third-party verification.
“2M context” does not match the 1,048,576 input tokens collected from the Google API model page; use the current specifications for the target endpoint.
Fix the model snapshot, API endpoint, thinking level, temperature, and max output.
Use the same versions of ARC AGI 2, IF-Evals, BFCL v3, LiveCodeBench, SWE Bench Lite, and BIRD-CRITIC; record each sample's input, output, tool calls, and failure reason.
Report all six scores and confidence intervals; do not let an average conceal the task gap between 92.3% and 32.5%.
Show the Google 77.1% baseline alongside these results and explain dataset/harness differences in the report.
Gemini 3.1 Pro
