Gemini 3.8 Flash · Media / benchmark · Independent measurement
The research note preserves an earlier Finance Agent v2 snapshot. The current leaderboard summary has changed; distinguish Partial Credit, All-Pass, and category results rather than reuse the historical rank.
The body preserves research collected on September 8, 2026. The editorial summary and conditions above were reviewed on September 20. References to “current” values, rankings, and dynamic metrics in these notes describe the original snapshot, not live results. Reopen the source before making a decision.
On the overall Vals AI Finance Agent v2 tasks, Gemini 3.8 Flash achieved 61.44% ± 0.13 Partial Credit, ranking No. 1 among the 58 systems listed on the page; its strict All-Pass score was 49.69% ± 0.42, ranking No. 2. It is suitable for evaluating an agent that retrieves, verifies, and completes multi-step financial analysis from public-company filings, but should not be interpreted as a measure of general intelligence or production investment-research accuracy.
Tasks suitable for observation: Quantitative and qualitative analysis of public-company filings, market analysis, earnings analysis, disclosure analysis, comparables, precedent transactions, adjustments, and financial modeling.
Tasks not suitable for direct inference: General question answering, non-financial domains, real-time trading decisions, agents without equivalent tools and document context, or the stability and compliance of production systems.
Applicable model version: Gemini 3.8 Flash on the Vals AI model page; do not extend this to Gemini 3.7 Flash, Gemini 3.8 Flash Cyber, or other providers/reasoning tiers.
Applicable client, agent, or API: The shared default harness for Vals AI Finance Agent v2; this is not an end-to-end score for the Gemini App or an arbitrary self-built agent.
Recommended parameters: The page shows Google provider, temperature 1, top-p/top-k DEFAULT, max output 65,536, and reasoning effort HIGH as its defaults; it also warns that different benchmarks may use different providers/parameters, so these page defaults must not be treated as a complete Finance Agent v2 run log when reproducing the test.
Target task: Simulate an entry-level financial analyst answering difficult questions based on public-company filings; questions typically require finding evidence across multiple filings or data sources, applying financial conventions, and keeping intermediate figures precise.
Shared default harness: Each agent can use six tools: edgar_search (SEC EDGAR API), web_search, parse_html_page, retrieve_information, calculator, and price_history.
Time limit: 2 hours per question; timeouts receive 0 points.
Runs and statistics: Each model was run 3 times; the page reports the mean and standard error (SEM) across the three runs.
Scoring: Each question consists of weighted checks and includes dealbreakers (critical facts/numbers). Partial Credit is the per-question average weighted by severity and gated by dealbreakers; if a critical item fails, the question receives 0. All-Pass is 100% only when every check passes; otherwise it is 0.
Evaluators: The Vals AI page lists three LLM evaluators: GPT-5.4, Gemini-3.1-Pro, and Claude Sonnet 4.6.
Question design: Questions require determinate answers, multi-source synthesis, domain specificity, and forensic precision (critical facts may be buried in footnotes, MD&A qualifiers, or accounting-policy disclosures).
Finance Agent v2 contains 927 expert-reviewed questions, divided as follows:
| Data section | Size | Availability |
|---|---|---|
| Public | 27 | Open sample and agent harness accessible |
| Private Validation | 450 | Requires authorization/permission |
| Test | 450 | Kept private; published page results are based only on this section |
The questions are divided into 9 analysis categories: General Qualitative Analysis, General Quantitative Analysis, Market Analysis, Comparables, Precedents, Adjustments, Earnings Analysis, Disclosure Analysis, and Financial Modeling. The page shows the highest accuracy by category ranging from 83.4% for General Qualitative to 34.5% for Financial Modeling, illustrating the benchmark's difficulty range from retrieval and summarization to multi-step modeling; these are the results of the leading model in each category, not Gemini 3.8 Flash's category-by-category scores.
| Scoring metric | Accuracy | Page rank | Cost per test | Input/output pricing | Page runtime |
|---|---|---|---|---|---|
| Partial Credit | 61.44% ± 0.13 | 1 / 58 | $2.00 | $1.50 / $7.50 per million tokens | 3 min 21 sec |
| All-Pass | 49.69% ± 0.42 | 2 / 58 | $2.00 | $1.50 / $7.50 per million tokens | 3 min 21 sec |
The Vals AI model page also lists Finance Agent (v2) at 61.44% ± 0.13, 1/58; the Vals Index in that page's overview is a separate aggregate metric and cannot replace the dedicated Finance Agent v2 score.
Strong performance as a professional finance agent: In Vals AI's Finance Agent v2 Overall table, Gemini 3.8 Flash ranks first with 61.44% Partial Credit; the adjacent second-place model, Muse Spark 1.2, scored 60.60% ± 0.28, a very small gap that does not justify claiming broad cross-task leadership.
Strict accuracy remains limited: All-Pass falls to 49.69% and ranks second; the gap between Partial Credit and All-Pass indicates that the model may capture key points while still missing peripheral checks or precise figures.
Cost should be assessed across the full test: The page records $2.00 per test and separately lists input/output pricing of $1.50/$7.50 per million tokens; this is not a fixed bill for any API request and does not mean costs remain unchanged after switching providers.
Cannot fully reproduce every question: The public section contains only 27 questions; the 450 Test questions remain private, so the published 61.44%/49.69% cannot be recalculated from the public set alone.
The harness has a major effect: The six tools, document retrieval, retrieval method, calculator calls, and stopping conditions all contribute to the score; this score should not be applied to Gemini 3.8 Flash outside that harness.
The scoring is not a single accuracy rate: Partial Credit is affected by dealbreakers and severity weighting, while All-Pass is a binary measure requiring every check to pass; they should not be conflated into one “accuracy” figure.
Dynamic snapshot: The page is marked UPDATED 9/5/2026; the model list, prices, rankings, and model availability may change, and the figures in this article correspond only to the page snapshot collected on 2026-09-08.
Limited question scope: The questions focus on public-company filings and financial-analyst work; they do not cover visual understanding, general coding ability, Chinese-localized investment research, compliance judgments, or actual investment returns.
Page parameters are not a complete run log: The default provider/temperature/reasoning settings on the model page cannot prove that every benchmark run used exactly the same configuration; the Vals AI benchmark page and the target run logs should be treated as authoritative.
Fix the Vals AI Finance Agent v2 page version, the exact model name Gemini 3.8 Flash, provider, temperature, top-p/top-k, max output tokens, reasoning effort, and pricing snapshot.
Use the six tools in the shared default harness and the two-hour per-question limit; retain the trace of every search, page parse, information retrieval, calculator, and price-history call.
First use the public 27 questions to verify the integration, then request access to the same Test split; do not claim to reproduce the page's 450-question results using public or self-authored questions.
Run each question 3 times, and calculate Partial Credit and All-Pass using the same check weights, dealbreaker gating, severity rules, and three-evaluator setup, reporting the mean and SEM.
Also record cost per test, input/output tokens, total runtime, timeouts, tool errors, and fallback behavior; if the provider or harness differs, name it as a new experiment and do not merge it with the ranking on this page.
Original Vals AI Finance Agent v2 page: https://www.vals.ai/benchmarks/fabv2; the page shows UPDATED 9/5/2026, 927 expert-reviewed questions, the shared harness, 6 tools, a 2-hour limit, 3 runs, and the definitions of Partial Credit/All-Pass.
Overall table on the same page: Gemini 3.8 Flash achieved Partial Credit of 61.44% ± 0.13, 1/58, $2.00/test, 3m21s; after switching to All-Pass, it was 49.69% ± 0.42, 2/58.
Original Vals AI model page: https://www.vals.ai/models/google_gemini-3.8-flash; the page lists the model release date, Google provider, token pricing, Finance Agent (v2) at 61.44% ± 0.13 and 1/58, and the page's default hyperparameters.
Vals AI summarizes Finance Agent v2 as “Evaluating agents on core financial analyst tasks.” This accurately limits the test subject: it is a benchmark of professional financial-analysis workflows, not an overall assessment of Gemini 3.8 Flash's general capabilities.
The figures, task set, reasoning tier, and client conditions apply only to the listed source and collection snapshot. Different versions, harnesses, or providers must not be compared directly; undisclosed parameters remain unknown.
For a reproduction, fix the model version, provider or client, reasoning tier, tools, task-set version, sample count, and collection date, and record failures, retries, and human corrections. Full steps are in the source notes below.
Vals AI · Vals AI · Original publication date 2026-09-05 · Site edit date 2026-09-20
Open original sourceGemini 3.8 Flash
Download the Tabbit client to check model access
