DeepSeek-V4.1-Flash Official Model Card Benchmarks: Agent Strengths and Harness Boundaries

One-sentence takeaway

The official model card shows that DeepSeek-V4.1-Flash is a multimodal MoE with a 552B backbone and 8B (prefill) / 16B (decode) activated per token; with the specified maximum reasoning effort and Agent harness, it scores 90.6 on Terminal-Bench 2.1 and 74.2 on DeepSWE v1.1, both above DeepSeek-V4-Flash in the table. Those numbers demonstrate the combined performance of the “model + harness + parameters” configuration and cannot be generalized as uniform capability across all clients or ordinary conversational tasks.

Use cases

• Tasks it is suitable for evaluating: Code Agents, terminal operations, repository modifications, tool calls, automation, and comparisons with the reference models listed in the model card under fixed evaluation conditions; multimodal capability can first be screened with MMMU-Pro, CVBench, DocVQA, and RefCOCO-avg.

• Tasks it is not suitable for extrapolating to: Generalizing official Agent scores to arbitrary Agent frameworks, chat products, production systems, real-world long-context business tasks, or safety-critical scenarios; results from different harnesses cannot be treated directly as a fixed score for the same model.

• Applicable model version: The open weights for deepseek-ai/DeepSeek-V4.1-Flash on Hugging Face; API versions, product endpoints, and later revisions require separate verification.

• Test environment or client: The Hugging Face weights and local inference method listed in the model card; Code Agent results use DeepSeek Harness Minimal mode, DeepSWE v1.1 separately uses the mini-SWE harness as required by its official setup, SEC-Bench Pro uses the Claude Code harness, and the vision Agent uses the Claude Code harness.

• Reasoning tier and parameters: The entire Instruct table uses the local-model evaluation setting reasoning_effort=100; temperature=1.0 and top_p=0.95. This is not the parameter syntax for hosted APIs, which use string tiers such as low/high/max. The model card recommends top_p=0.95 or 1.0, a 1M-token context window, and max_tokens ≥ 256K.

Evaluation method

The official results are divided into Base Model, Instruct Model, and comparisons across different Agent scaffolds.

All Base Models are evaluated in the official internal framework under the same evaluation settings; the model card treats scores within 0.3 of one another as equivalent. Shot counts vary by benchmark in the tables, so scores from different benchmarks cannot be treated as being on the same scale.

The Instruct Model supports continuous reasoning effort from 1 to 100. The model card explicitly states that the entire table below uses the maximum setting, reasoning_effort=100, with sampling at temperature=1.0 and top_p=0.95. It first summarizes the Code Agent benchmarks as using DeepSeek Harness Minimal mode with a 1M-token context, then explicitly lists the exceptions: DeepSWE v1.1 uses mini-SWE to follow the official setup, while SEC-Bench Pro uses Claude Code. Reproduction should select the harness according to each benchmark's exception notes. Chartography, BabyVision, and ZeroBench use the Claude Code harness with a 512K-token context; Agent's Last Exam and AutomationBench use their respective official scaffolds. The model card says that all Agent evaluations use temperature=1.0 and top_p=0.95.

In the scaffold comparison, DeepSWE v1.1 takes N=8 samples per question and Terminal-Bench 2.1 takes N=3 samples per question; the evaluations use Linux containers, max_steps=500, a 1M context limit, and the same temperature=1.0 and top_p=0.95 sampling. This Terminal-Bench 2.1 evaluation group does not allow network access.

Key results

In the model card's Agent comparison table, DeepSeek-V4.1-Flash scores higher than DeepSeek-V4-Flash in the same table on Terminal-Bench 2.1 (90.6 vs 82.7), Terminal-Bench 3.0 (30.0 vs 7.6), Terminal-Bench 4.0 (31.2 vs 7.0), DeepSWE v1.1 (74.2 vs 54.4), CyberGym (88.1 vs 76.7), and AutomationBench (54.8 vs 37.7). HLE's 36.8 and 37.8 with † use different definitions and cannot be compared directly; the scores for the same text-only subset are 39.1† and 37.8†.

The model card also reports results for the same model across scaffolds: DeepSWE v1.1 ranges from 69.8 with Claude Code to 74.2 with mini-SWE, while Terminal-Bench 2.1 ranges from 84.1 with Codex to 90.6 with DSH Minimal. This spread is enough to show that the Agent scaffold changes the observable score.

Raw data

The following numbers are transcribed from the official model card; — means the model card does not provide a result for that column, and † is the model card's footnote marker.

Base Model

Instruct Model: frontier comparison (maximum reasoning effort)

† Official footnote: HLE text-only subset.

Across Agent scaffolds

Conclusions and limitations

1. The strongest signal in the official results is within the Agent task bucket. Under the fixed settings listed in the model card, V4.1-Flash scores higher than V4-Flash on Terminal-Bench, DeepSWE, CyberGym, and AutomationBench, and has a Codeforces rating of 3471; this supports considering it as a candidate model for Code Agents and terminal tasks.

2. Benchmarks are not a unified capability score. The Base table uses different shot counts and metrics; the Instruct table mixes Pass@1, Pass@5, Resolved, Rating, Score, and LLM-Judge, so it cannot be sorted across columns to produce a single “overall capability” score.

3. The harness changes the result. For the same model, scaffold results range from 65.5 to 74.2 on DeepSWE v1.1 and from 84.1 to 90.6 on Terminal-Bench 2.1. Any repeat evaluation must fix the Agent scaffold, tool protocol, step count, context, network access, and sampling parameters.

4. Maximum reasoning effort limits extrapolation. All official Instruct results use reasoning_effort=100, which does not represent performance at lower effort or the default setting; the model card provides no curves showing how each benchmark changes with reasoning effort.

5. Long-context support does not equal long-task performance. The model card gives a 1M-token context window and a LongBench-V2 Base result of 45.2, but provides no business data, long-session failure samples, or complete variance, so it cannot guarantee any arbitrary 1M-token workflow.

6. The official table does not provide all information required for independent verification. The page does not publish the complete inputs, outputs, repeated experiments, confidence intervals, or all harness implementations for each benchmark in the table; some comparison columns are external model names, and the evaluation sources and run details cannot be fully reconstructed from the table alone.

7. Efficiency and capability require separate validation. The model card states a 552B total backbone, 8B/16B activated parameters, and a 1M context, but does not provide standardized hardware, quantization, concurrency, throughput, or latency data in this results table, so production cost cannot be inferred directly from parameter scale.

Reproduction notes

1. Fix the commit of the Hugging Face model repository, weight precision, tokenizer, inference engine, and prompt encoding; the model card says this version has no Jinja chat template, and the repository provides encoding/encoding.py as a reference implementation.

2. For Instruct reproduction, start with the model card parameters: reasoning_effort=100, temperature=1.0, top_p=0.95, and a 1M context; also record max_tokens, and do not mix results with different context limits.

3. Run Code Agents separately with the harnesses specified by the model card: DeepSeek Harness Minimal mode, mini-SWE, Claude Code, and each benchmark's official scaffold; for Terminal-Bench 2.1 reproduction, disable network access and record the Linux container, N, max_steps=500, and tool-call logs.

4. Save the raw input, model output, patch, test results, failure reason, tool-call count, and token usage for every task; calculate metrics such as Pass@1, Pass@5, Resolved, and Rating separately, without merging them into an overall score.

5. First reproduce the public DeepSWE v1.1 and Terminal-Bench 2.1 results, then repeat the same task sets in the target production Agent; report the difference between the official results and those from your own harness, and mark the collection date and model commit.