On one Atomic Agent coding task, both Qwen 4 Max's high and xhigh levels passed all 59 hidden tests in all three runs. High took about 25 minutes on average and cost about $0.65 on average across the two fully recorded runs. Xhigh did not increase the number of tests passed, but took nearly twice as long and cost more.
What this can help assess: Completion, runtime, and API cost at high versus xhigh for a single codebase task.
What this cannot be generalized to: Other codebases, non-coding tasks, overall long-task success rates, or the model's capability ceiling.
Applicable model version: Qwen 4 Max (OpenRouter qwen/qwen4-max; the article says Alibaba was the only provider available on that route at the time).
Test environment or client: The version included in Atomic Agent 0.7.0; OpenRouter API; 3 runs per reasoning level, each with a 60-minute time limit.
Reasoning levels and parameters: high (Alibaba's default, explicitly specified in the test) and xhigh; other sampling parameters were not specified.
The task used a stock library of about 1,400 lines with 13 built-in defects and one missing feature, scored against 59 hidden tests. Each reasoning level was run three times, with a 60-minute limit per run; all six Qwen 4 Max runs took place simultaneously. The Qwen 3.8 Max comparison data came from a separate batch of tests on November 9, using the same task and build version.
| Model and level | Test results | Runtime | Cost |
|---|---|---|---|
| Qwen 4 Max high | 59/59 in all 3 runs | 23.7, 24.1, and 27.2 minutes | Mean of the two complete records: about $0.65 ($0.62, $0.68); the third record is incomplete |
| Qwen 4 Max xhigh | Reached 59/59 in all 3 runs | Two runs: 47.3 and 47.4 minutes; the third reached the 60-minute limit | Recorded single-run costs of $0.84–$1.28; requests are missing from the records, so actual costs were higher |
| Qwen 3.8 Max | The three completed batches scored 26, 58, and 24/59, respectively; a separate early-aborted batch reached 59/59 | All three stopped at 60 minutes; the early batch ran for 45 minutes | Full costs not provided |
The authors report 34K and 41K reasoning tokens for the two fully recorded high runs, and at least 78K–133K for xhigh. The task passed all tests at high; switching to xhigh did not increase the number passed.
high: Passed 59/59 in every run; 76 steps on average; complete bills for two runs were $0.62 and $0.68.
xhigh: Each run is recorded as 59/59 in the table; 75 steps. The third run was stopped by the 60-minute limit, but the code had already passed all tests at that point.
Comparison: The authors say Qwen 3.8 Max did not complete the task in the three completed 60-minute batches; an earlier batch passed all tests before it was stopped because the OpenRouter balance ran out.
This case shows that high was sufficient to pass the tests on this task, while xhigh brought longer runtimes and higher costs. The sample covers just one task and three runs per level, so it cannot establish general success rates or rank model capabilities. All six Qwen 4 Max runs shared one machine and one API key; the comparison runs were conducted on other dates, and Kimi's runs were sequential, so cross-model runtime comparisons are only rough references. Some cost records are also missing: the third high run was not fully recorded, and the xhigh figures are a recorded range rather than complete final bills.
The article says to select the model using Atomic Agent, OpenRouter, and qwen/qwen4-max, then set reasoning.effort to xhigh in the provider configuration; high is the default level. The article does not disclose the stock library task input or the 59 hidden tests, so a similar task can be rerun with a comparable environment and reasoning level, but the original test cannot be reproduced exactly. The article also notes that running all six jobs in parallel may affect runtime.
Qwen4-Max