Gemini 3.5 Flash · Media / benchmark · Independent measurement
Appwrite Arena's May 20, 2026 run reports freeform rising from 77.5% to 91.9% after loading the Appwrite Skill, showing that documentation context changes agent results.
Unverified: the original source could not be rechecked. Historical figures below are not current verified results.
Appwrite's public Arena results show that Gemini 3.5 Flash has advantages in MCP/tool use, multimodality, and speed, while loading the Appwrite Skill raises its freeform score from 77.5% to 91.9%, indicating that documentation context has a significant effect on its current knowledge coverage.
Suitable tasks: SDK/API agents, MCP tool calls, documentation-driven backend development, and throughput-first multimodal and short-iteration workflows.
Unsuitable tasks: difficult academic reasoning, guessing the latest APIs without domain documentation, or generalizing Appwrite-specific results to all software engineering tasks.
Applicable model versions: Gemini 3.5 Flash; the article compares Gemini 3 Flash, Gemini 3.1 Pro, Claude Sonnet 4.6, Claude Opus 4.7, GPT-5.5, and others.
Applicable clients, agents, or APIs: Gemini API, Appwrite API/Docs MCP, and Appwrite Skills; the article recommends Cursor, Google AI Studio, Google Antigravity, or the API directly.
Recommended reasoning tier and parameters: The article says the main Google figures use high thinking; Arena runtime parameters and temperature are not fully listed, so they cannot be fixed based on this article.
Benchmark: Appwrite Arena, described in the article as an open-source benchmark covering 191 questions across 9 Appwrite service categories; each model runs under both “with Skill” and “without Skill” conditions.
Task types: Appwrite Foundation, Auth, Databases, Functions, Storage, Sites, Messaging, Realtime, and CLI; includes multiple-choice and freeform tasks.
Comparison models: GPT 5.5, Claude Opus 4.7, Qwen 3.6 Plus, Kimi K2.6, DeepSeek V4 Flash, Gemini 3.1 Pro, and others.
Evaluation dimensions: overall, MCQ, freeform, cost, and duration; the article links to the Arena GitHub repository and public leaderboard.
“With Skills” puts the relevant Appwrite documentation into the context; “Without Skills” uses only the model's built-in knowledge.
The article reports a run on 2026-05-20; it does not disclose a complete snapshot of the question set, the OpenRouter request body, random seed, temperature, per-question token counts, or evaluator prompt.
The article also cites the Google model card and Artificial Analysis; these should be understood separately from the Appwrite Arena results.
| Benchmark | Gemini 3.5 Flash | Main results |
|---|---|---|
| Terminal-Bench 2.1 (Terminus-2) | 76.2% | GPT-5.5 78.2%, Gemini 3.1 Pro 70.3%, Opus 4.7 66.1% |
| SWE-Bench Pro | 55.1% | Opus 4.7 64.3%, GPT-5.5 58.6%, Gemini 3.1 Pro 54.2% |
| MCP Atlas | 83.6% | GPT-5.5 75.3%, Opus 4.7 79.1%, Gemini 3.1 Pro 78.2% |
| Toolathlon | 56.5% | GPT-5.5 55.6% |
| OSWorld-Verified | 78.4% | GPT-5.5 78.7%, Opus 4.7 78.0%, Gemini 3.1 Pro 76.2% |
| Finance Agent v2 | 57.9% | GPT-5.5 51.8%, Opus 4.7 51.5%, Gemini 3.1 Pro 43.0% |
| CharXiv Reasoning | 84.2% | GPT-5.5 84.1%, Gemini 3.1 Pro 83.3% |
| MMMU-Pro | 83.6% | GPT-5.5 81.2%, Gemini 3.1 Pro 80.5% |
| Humanity’s Last Exam | 40.2% | Opus 4.7 46.9%, GPT-5.5 41.4%, Gemini 3.1 Pro 44.4% |
| Condition | Overall | MCQ | Freeform | Cost | Duration |
|---|---|---|---|---|---|
| With Skills | 96.20 | 96.90 | 91.90 | $3.78 | 20 minutes |
| Without Skills | 90.70 | 92.90 | 77.50 | $1.14 | 13 minutes |
The article calculates that the Skill increases freeform performance by 14.4 percentage points; under the with-Skills condition, Gemini 3.5 Flash is faster than most of the higher-scoring models in the same table, but GPT-5.5 and Claude Opus 4.7 still have higher overall scores.
Intelligence Index: 55.3, ranked #7 out of 147 models.
Speed: 278 output tokens/s, ranked #2 in its price tier.
Full-index output volume: 73M tokens; total evaluation cost: $1,552.
The article reports a hallucination measure of 61% and emphasizes that token volume and price together determine the cost of demanding reasoning tasks.
Gemini 3.5 Flash is suited to agents where speed, tools, and contextual documentation matter more than purely built-in knowledge. The most reusable conclusion from the Appwrite-specific experiment is not the absolute ranking, but that “providing the current API documentation to the model as a Skill” significantly improves freeform tool tasks; for difficult reasoning and correctness-first coding, a stronger model should remain available as a comparison point or upgrade path.
Appwrite Arena is a domain-specific benchmark; its results do not represent general coding, mathematical, or open-domain capabilities.
Although the article gives the number of questions, Skill conditions, cost, and duration, it does not disclose the complete request configuration, scoring prompts, random seed, or original per-question results.
The current README in the Appwrite GitHub repository shows a different version with 70 questions (57 multiple-choice + 13 freeform, across 7 categories), and its model list does not include Gemini 3.5 Flash; this differs from the article's claimed 191 questions/9 categories. Reproduction therefore requires pinning the benchmark commit or data snapshot corresponding to the article.
Artificial Analysis and Appwrite Arena use different scores, prices, and task definitions; they cannot be combined into a single capability score.
Pin the Appwrite Arena question-set version, Skills files, model ID, OpenRouter route, thinking level, and evaluator.
Run Gemini 3.5 Flash under both with-Skills and without-Skills conditions, saving each response, MCQ scores, freeform evaluator outputs, token counts, cost, and duration.
Repeat at least 3–5 times and report the mean, variance, and failure categories; a single overall score is insufficient to determine production stability.
Also run an A/B test on the target project's own SDK/MCP task set to compare whether the benefit of documentation injection persists.
The article summarizes Gemini 3.5 Flash as “speed is the column where it dominates,” while acknowledging that speed is not the same as coding intelligence. The most reusable engineering observation is that the model relies noticeably on Skills documentation for Appwrite freeform tasks.
The figures, task set, reasoning tier, and client conditions apply only to the listed source and collection snapshot. Different versions, harnesses, or providers must not be compared directly; undisclosed parameters remain unknown.
For a reproduction, fix the model version, provider or client, reasoning tier, tools, task-set version, sample count, and collection date, and record failures, retries, and human corrections. Full steps are in the source notes below.
Appwrite Blog / Appwrite Arena · Atharva Deosthale · Original publication date 2026-05-20 · Site edit date 2026-09-20
Open original sourceGemini 3.5 Flash
Download the Tabbit client to check model access
