On Plotly's 43-question data analytics benchmark, Qwen4 Max answered 35 questions correctly on a single run (81%, B), at a cost of $2.29 and a median of 40 seconds per question. It handled both ambiguous questions and requests it could not complete correctly, but lost points on data quality checks, statistical pitfalls, and hidden patterns.
Tasks this can help assess: Performance on multi-turn tool use, chart analysis, clarifying ambiguity, and declining requests it cannot fulfill, using Plotly's Dash data analytics Agent and its synthetic wind power data warehouse.
Tasks this should not be generalized to: Other datasets, business databases, different Agents or API configurations, or overall model capability rankings. The benchmark uses a private question set, and the results represent a single run only.
Model version covered: Qwen4 Max, as named in the article; no API model ID or snapshot identifier is given.
Test environment or client: Plotly Studio Embedded (an analytics Agent running in a Dash app), accessed through OpenRouter. Plotly says all tested models used the same harness and scoring method.
Reasoning tier and parameters: Not specified.
Plotly used its PlotlyDataAnalyticsBench, a set of 43 questions run through a multi-turn tool-using Agent for data analytics queries and chart generation. The benchmark contains 20 standard analysis questions, 5 ambiguous questions, 5 requests that cannot be completed, 3 data quality issues, 5 statistical pitfalls, and 5 hidden-pattern questions. The questions use a fictional wind power operator's data warehouse. Plotly says the question set is private and available to researchers.
This article reports one run of Qwen4 Max. Plotly notes that repeated runs of other models have varied by as many as 3 points, so single-run scores that differ by 1–2 points should be treated as a near tie.
Overall score: 35/43 (81%, B). By category: standard analysis 19/20, ambiguous questions 5/5, requests that cannot be completed 5/5, data quality 1/3, statistical pitfalls 3/5, and hidden patterns 2/5.
Of the 8 incorrect answers, one had an incorrect final number because of precision loss in a time calculation. The other errors were 2 data quality issues, 2 statistical pitfalls, and 3 hidden-pattern questions.
The model was called about 6 times per question and generated about 3,200 tokens per question; approximately 92% of the input tokens read were from the provider's cache. The full benchmark took about 47 minutes, with a median of 40 seconds per question.
The single-run cost for the full benchmark was $2.29. At the article's assumed usage of 10 questions per person per day, 250 workdays per year, and 1,000 people, the estimated cost is about $133 per person per year; this estimate uses the stated workload and OpenRouter prices at the time of the run.
Comparisons listed in the same article: Qwen 3.8 27B scored 38/43 at $1.71 and a median of 53 seconds per question; Kimi K3 scored 35/43 at $5.01 and 43 seconds; Hy4 scored 42/43 at $2.11 and 117 seconds. Each model's result is from a single run.
The article reported Qwen4 Max pricing at the time as $1.50/$4.50 per million input/output tokens. These prices and the single-run cost reflect data at publication time.
| Metric | Qwen4 Max |
|---|---|
| Overall score | 35/43 (81%, B) |
| Standard analysis | 19/20 |
| Ambiguous questions | 5/5 |
| Requests that cannot be completed | 5/5 |
| Data quality | 1/3 |
| Statistical pitfalls | 3/5 |
| Hidden patterns | 2/5 |
| Single-run cost for the full benchmark | $2.29 |
| Median time per question | 40 seconds |
| Model calls per question | About 6 |
| Output tokens per question | About 3,200 |
| Share of input tokens served from cache | About 92% |
In this test, Qwen4 Max handled most standard questions and all ambiguous questions and requests it could not complete correctly. It lost points on finding data quality issues, identifying misleading statistics, and searching for hidden patterns. Compared with Qwen 3.8 27B in the article, Qwen4 Max scored 3 points lower and cost more, but had a faster median time. This conclusion is limited to Plotly's harness and question set and each model's single run; it does not establish general data analytics capability or a stable cross-model ranking.
Reproduction requires access to Plotly's private PlotlyDataAnalyticsBench, its synthetic wind power data warehouse, scoring rules, and Plotly Studio Embedded harness. The public article does not provide the complete questions, data, answer key, run logs, or Qwen4 Max API model ID, reasoning tier, and request parameters, so the original score cannot be independently rerun and confirmed using the public material alone. The article says researchers can contact Plotly to request access.
Qwen4-Max