SimpleBench: everyday reasoning and language traps

Historical snapshot notice

The body preserves research collected on September 8, 2026. The editorial summary and conditions above were reviewed on September 20. References to “current” values, rankings, and dynamic metrics in these notes describe the original snapshot, not live results. Reopen the source before making a decision.

One-sentence takeaway

The current SimpleBench leaderboard lists Gemini 3.8 Flash at No. 4 with 82.4% (AVG@5); it is below the highest human score of 95.4% and Claude Fable 5.1's 86.6%, and also below the human baseline of 83.7% (by 1.3 percentage points). This indicates that on this benchmark's everyday physics, time, social common-sense, and language-trap questions, it is close to but has not yet reached the average human level in this sample; the result cannot be extrapolated into a ranking of general reasoning ability.

Use cases

• Tasks suitable for observation: Spatial and temporal reasoning, social intelligence, and real-world judgments involving misleading wording or wordplay in English six-option multiple-choice questions.

• Tasks not suitable for direct inference: Chinese understanding, visual reasoning, coding, long-horizon agents, specialized knowledge, or production-system success rates.

• Applicable model version: The Gemini 3.8 Flash explicitly identified by the official leaderboard; do not extend this to Gemini 3.8 Flash Cyber or other providers, clients, or reasoning tiers.

• Applicable test subject: SimpleBench's standard text multiple-choice questions; this is not an end-to-end evaluation of the Gemini App or API.

Benchmark and method

• Question count: The technical report defines 204 independent questions, each with 6 options; the website summarizes this as 200+ questions. The questions were written manually by the team and reviewed by internal members and 4 PhD-level experts.

• Coverage: Approximately 40% spatial reasoning, 10% temporal reasoning, 25% social intelligence, and 25% adversarial language/trick questions; categories may overlap.

• AVG@5: Each model is run independently 5 times; the proportion correct is calculated for each run, then the arithmetic mean of the 5 accuracies is taken:

  AVG@5 = (accuracy on run 1 + … + accuracy on run 5) / 5

  There is also MAJ@5, where a question counts as correct only if it is answered correctly at least 3 times out of 5; the current website leaderboard's column is Score (AVG@5). It does not publish Gemini 3.8 Flash's individual-run scores, MAJ@5, or confidence interval.

• Sampling settings: The website's settings panel shows temperature 0.7 and top-p 0.95 (except for the o1 series). The historical technical report also states that all models use a unified instruction, with the standard prompt requiring the output Final Answer: A-F; a website footnote mentions that an engineered prompt optimized for the benchmark may be tried, but the specific prompt for the current Gemini 3.8 Flash entry is not disclosed.

• Scoring: Final Answer: X is extracted from the model output and compared with the correct answer letter; a parsing failure raises an error. The official code enables majority voting only when num_responses > 1, with one response by default, so the code itself cannot directly produce the website's AVG@5.

Gemini 3.8 Flash results

The leaderboard data file marks this entry as dateAdded: 2026-09-03, with Google as the organization:

The leaderboard provides only point estimates and relative rankings; adjacent models do not necessarily share the same timestamp, provider, or complete run configuration, so rank differences must not be treated as statistical significance.

Human sample

• Sample: 9 native English-speaking participants with at least a high-school level of mathematics.

• Assignment: Each person randomly completed 25 questions, collectively covering all 204 questions at least once; no one answered all 204 questions.

• Results: The average score across the 9 participants produced the 83.7% Human Baseline; the current website leaderboard separately lists a 95.4% Highest Human Score.

• Test instructions: Participants were asked to spend about 3 minutes per question, could use pen and paper, and were warned that questions might contain wordplay, spatial transformations, and social-common-sense traps.

Limitations and reproduction boundaries

• The public questions are insufficient to recalculate the leaderboard: After the current public simple_bench_public.json was opened and parsed from the official page, it contained only 10 questions, question_id 1–10; the remaining questions are private. The 10-question result cannot be presented as a reproduction of 82.4%.

• The official code cannot directly reproduce this model entry: The current GitHub project still maps only an old model list and has no Gemini 3.8 Flash; run_benchmark.py defaults to num_responses=1, and the project's current commit history is from 2024, predating the entry's 2026-09-03 leaderboard date.

• The run conditions are incomplete: The leaderboard does not publish Gemini 3.8 Flash's exact API model ID/snapshot, provider, complete prompt, five original outputs, individual scores, retry and failure-handling logs, token counts, or confidence intervals. Therefore, the official leaderboard point estimate of 82.4% can be checked, but each step of its run cannot.

• The human baseline has limited representativeness: The sample contains only 9 people and does not control for IQ or general reasoning ability; voluntary participation, English proficiency, response time, and fatigue may all affect 83.7%. This baseline is not a population-level human average.

• The benchmark's scope is limited: The questions are English, manually designed, six-option text questions focused on “the most realistic” everyday judgments and trap recognition; they do not cover open-ended answers, Chinese contexts, tool use, or real-world execution.

• Prompts may affect rankings: The official report shows that additional reminders about traps, real-world physics, and overconfidence can materially change older models' scores; the current entry does not disclose whether or how Gemini 3.8 Flash used that prompt, so 82.4% should not be treated as the model's intrinsic bare-model capability.

Reproduction recommendations

1. Fix the SimpleBench version, the complete 204-question private set, the original English questions, correct answers, Gemini 3.8 Flash's exact model ID/snapshot and provider; without the private set, report only an experiment on the public 10 questions.

2. Record the actual system prompt, temperature 0.7, top-p 0.95, max tokens, retry/parsing-failure handling, and every API output; do not use the current outdated model mapping as a substitute for 3.8 Flash.

3. Run the complete question set 5 times independently, calculating the 5 individual accuracies, AVG@5, and MAJ@5; also report each score and the number of missing/failed answers.

4. If comparing models, fix the same question set, prompt, sampling parameters, and number of runs; if using an engineered prompt, name the experiment separately and do not mix it with the standard leaderboard methodology.

5. Label local results as an “independent rerun” and do not merge them directly with the website's 82.4%; the website publishes only a leaderboard point estimate, and the original Gemini 3.8 Flash run conditions cannot be fully reconstructed.

Original evidence and data

• SimpleBench official homepage leaderboard: https://simple-bench.com/. The page lists Gemini 3.8 Flash at No. 4 with 82.4%, labels the column Score (AVG@5), and lists the human baseline of 83.7%, highest human score of 95.4%, and temperature/top-p settings.

• SimpleBench official technical report (PDF in a historical project commit): https://github.com/simple-bench/SimpleBench/blob/96fa7ada8cf046fb5332a37122f397754b128d7b/SimpleBench.pdf. The report defines the 204 questions, AVG@5/MAJ@5 across five independent runs, human sample, and limitations.

• SimpleBench official public data: https://raw.githubusercontent.com/simple-bench/SimpleBench/main/simple_bench_public.json. The current file contains 10 public questions, with question_id values 1–10.

• SimpleBench official reproduction code: https://github.com/simple-bench/SimpleBench/blob/main/run_benchmark.py, https://github.com/simple-bench/SimpleBench/blob/main/weave_utils/scorers.py. The code shows answer extraction, multiple-choice scoring, and majority-voting implementation, but has no model mapping for Gemini 3.8 Flash.

Source excerpt or observation (short quotation for compliance only)

SimpleBench describes itself as “Where Everyday Human Reasoning Still Surpasses Frontier Models.” This accurately limits what it measures to a set of English everyday-reasoning and language-trap questions, rather than a full assessment of all model capabilities.