Qwen3.7-Max: Official Complete Benchmarks and 35-Hour Autonomous Optimization Experiment

One-sentence takeaway

The official results show Qwen3.7-Max performing strongly on coding, MCP/Skills, reasoning, multilingual tasks, and long-horizon tool use, but the scores come from different harnesses; the most convincing reproduction path is to fix the task, tools, and verifier, then measure cost-per-success yourself.

Test environment

• Model: Qwen3.7-Max, compared with Opus-4.6 Max, K2.6 Thinking, GLM-5.1 Thinking, DeepSeek V4 Pro Max, and Qwen3.6-Plus.

• Agent frameworks: Claude Code, OpenClaw, Qwen Code, and custom tool frameworks; Qwen emphasizes decoupling the task, harness, and verifier to test cross-framework generalization.

• Terminal-Bench 2.0-Terminus: Harbor/Terminus-2, 5-hour timeout, 12 CPU/24GB RAM, temperature 1.0, top_p 0.95, top_k 20, max_tokens 80K, 256K context, averaged over 5 runs.

• SWE series: Internal Agent scaffold (bash + file-edit tools), temperature 1.0, top_p 0.95, 200K context.

• SkillsBench: OpenCode, 78 tasks, excluding 9 tasks dependent on external APIs, averaged over 5 runs.

• Kernel Bench L3: 50 questions, one isolated Docker container, one H100 80GB, network restricted, at most 500 tool calls; early stop after 100 consecutive calls without improvement.

Inputs/configuration

The official page publishes benchmark names, some harnesses, timeouts, and sampling settings, but not the complete prompt for every task, random seeds, all failed trajectories, or a downloadable run package. The 35-hour kernel case uses a T-head Zhenwu M890 PPU and an in-house evaluation script, so it cannot be treated as a general-purpose hardware test.

Results data

Coding and Agents

Reasoning, multilingual tasks, and long context

Long-horizon autonomous experiment

• Extend Attention kernel experiment: approximately 35 hours, 432 kernel evaluations, and 1,158 tool calls; geometric-mean speedup relative to the Triton reference implementation was 10.0x.

• Comparison: GLM-5.1 7.3x, Kimi K2.6 5.0x, DeepSeek V4 Pro 3.3x, and Qwen3.6-Plus 1.1x; the official page does not provide all details of the same hardware/tools.

• Reward-hacking monitoring: more than 80 hours of RL experiments and tens of thousands of calls; 13 new heuristic rules were added and 1,618 cheating cases were identified.

• YC-Bench business simulation: Qwen3.7-Max generated $2.08M in revenue and completed 237 tasks; this is an official simulated environment, not real-world enterprise ROI.

Conclusions

• Coding and tool use are the areas with the densest official evidence: SWE-Verified 80.4, MCP-Atlas 76.4, Skillsbench 59.2, and Kernel Bench L3 1.98x/96%.

• MRCR-v2 128K is 90.4; together with the 1M-level context positioning, this makes the model a candidate for long-document/long-horizon Agent testing, but a single retrieval score cannot be extrapolated to every document structure.

• The 35-hour experiment supports the claim that the model can maintain long-horizon progress under specific tools and verifiers; it does not mean every Agent harness can run unattended for 35 hours.

• The multi-benchmark tables show breadth of capability, but the judges, tools, and sampling configurations differ by task; comparisons should be split by task type.

Limitations

• Officially released data carries vendor-selection, internal-benchmark, and self-reported-result bias; there is no independent blind test.

• The tasks and evaluation details for internal benchmarks such as QwenWebDev, QwenClaw, CoWorkBench, and YC-Bench are incomplete, so they cannot be independently recomputed.

• Some scores were produced with different harnesses, judges, or environments; horizontal ranking across tables is not rigorous.

• The 35-hour kernel run, RL monitoring, and enterprise simulation environment are highly specific and cannot directly serve as production SLAs, investment returns, or safety guarantees.

Reproduction steps

1. Choose a public task set and fix the Qwen3.7-Max dated snapshot, temperature, top_p, thinking switch, tool versions, context, and timeout.

2. For Agent tasks, save the versions of the task, harness, and verifier; record tool calls, failures, retries, tokens, wall-clock time, and cost for every round.

3. For Terminal/SWE, repeat 5 runs first, then report the mean, variance, first-pass rate, and number of manual corrections.

4. Set a no-improvement stopping threshold for long-horizon tasks, and report “completion rate” separately from “runtime duration” to avoid promoting only the longest run.

5. Have at least one non-Qwen judge or independent code checker review the result, and compare cost-per-success rather than a single benchmark.

Source excerpt or observation (brief excerpt for compliance only)

The official page describes the model as “The Agent Frontier,” while also providing the harness and limitations for each benchmark; reproductions should retain these conditions.