Plotly's 43-question data analysis exam: Kimi K3.1 Pro scores 42/43

One-sentence takeaway

On its own PlotlyDataAnalyticsBench, Plotly asked the standard Kimi K3.1 Pro to answer 43 synthetic wind-power data questions, getting 42/43 (98%, A); it found all 5 hidden patterns and 3 data-quality issues, but one run cost $13.47 and took a median of 2.5 minutes per question, showing a clear trade-off among accuracy, proactive analysis, and cost and speed.

Use cases

• Tasks this can help assess: Data-analysis Agents with SQL or data-query tools; workflows that need to identify hidden patterns, data-quality issues, ambiguous questions, and unanswerable requests.

• Tasks this should not be generalized to: Overall success rates across all data-analysis tasks, real-world wind-power business data, general reasoning rankings, stable production costs, or latency.

• Model version covered: Standard Kimi K3.1 Pro; not Kimi K3.1 Pro Highspeed.

• Test environment or client: Plotly's private data-analysis exam; the model was called through OpenRouter, and the original article does not disclose the complete harness, tool schema, or execution logs.

• Reasoning level and parameters: The original article says K3.1 Pro uses the maximum reasoning intensity by default and always reasons, but does not disclose the complete request parameters for this run.

Evaluation method

Plotly continuously tests new models with the same exam. The data warehouse behind the question set contains synthetic wind-farm data designed to resemble data from real operators; the data and answers do not appear in real-world sources. The exam itself remains private: the author says this is to keep the answers out of training sets and that access can be requested from Chris Parmer.

This record covers one complete run of the standard K3.1 Pro through OpenRouter on the release date. The original article does not disclose the text of the 43 questions, answer key, complete system prompt, tool definitions, each query, random seed, or raw output. This article therefore does not rewrite the exam as a directly reusable prompt or claim that readers can fully reproduce the original score.

The exam result is recorded using this formula:

Total score = number of correct answers / 43 = 42 / 43 = 97.67% ≈ 98%

The original article places the result on Plotly's leaderboard of 34 models and compares it with results from Kimi K3, Grok 4.7, GPT-6.1 Sol, GPT-6 Astra, and MiniMax M3.1 Flash, among others. These cross-model figures come from Plotly's own exam and harness and cannot be directly combined with scores from other benchmarks.

Key results

On Plotly's 34-model leaderboard, K3.1 Pro's score of 42 ties Grok 4.7 for third place; GPT-6.1 Sol and GPT-6 Astra scored 43. Plotly calls this the best score from a Chinese lab model it has tested, and it is also higher than MiniMax M3.1 Flash's score of 41. This ranking is only a snapshot of this exam as of the original article's publication.

Raw data

Main results and proactive analysis

• K3.1 Pro got 42/43 correct, displayed in the original article as 98% (A).

• It found all 5 patterns in 5/5 Hidden needles that had been planted in the data without being stated in the questions.

• In 3/3 Data quality, it identified duplicate rows, the missing final day of a month, and the distinction between a turbine that was retired and had no data and one that genuinely produced no power.

• In a question asking it to calculate a reliable annual failure rate for a retired turbine and then use it to plan the budget for the entire fleet, it pointed out that the turbine's history was too short and instead provided a fleet-level failure rate.

• It scored 20/20 and 5/5 on the analytics and ambiguous questions respectively; unanswerable requests were 5/5.

• It scored 4/5 on statistics traps, the same as Kimi K3.

The only error

The only incorrect answer involved turbine-days with no firmware. The question asked it to compare days with firmware FW-2.4.1 against days with other firmware. The model chose a rigorous test for the FW-2.4.1 group and calculated the mean correctly; however, it counted the 2,775 turbine-days before the turbines had any firmware installed in the “other firmware” group, while also claiming that every turbine-day had firmware.

This made the mean for the “other firmware” group about 0.0003 too high, just beyond Plotly's tolerance. The error was an inconsistency between the grouping condition and the model's own description, rather than a wholesale arithmetic error.

Calls, queries, and duration

• About 7.5 model calls per question on average, close to the average among models Plotly has tested.

• On hidden-pattern questions, the model ran multiple queries in a single step, at about 73 tool calls per question; GPT-6.1 Sol used about 18 by comparison.

• Median time per question was about 2.5 minutes, making it the second slowest on the leaderboard, faster only than MiniMax M3.1 Flash.

• About 90% of the time was spent waiting for the model response and about 10% running SQL; the slowness mainly came from generating reasoning content, rather than query execution.

• On the release date, generation through OpenRouter was about 64 tokens/s, slightly below the roughly 72 tokens/s median among tested models and close to Grok 4.7's 67 tokens/s.

• It generated about 16,500 output tokens per question, about 62% of which was reasoning content; the output volume was about 5 times Kimi K3's and 7 times GPT-6.1 Sol's.

• On direct analysis questions, about 99% of the time was spent on the model response; query execution accounted for about one-quarter of the time on hidden-pattern questions.

Cost

The original article uses OpenRouter pricing: $2.00 per million tokens for input, $10.00 per million tokens for output, and $0.20 per million tokens for cached input. The full exam cost was $13.47, or about $0.31 per question:

Average cost per question = $13.47 / 43 ≈ $0.313

The model read about 440,000 tokens per question, of which 92% were cached input; it output about 16,500 tokens. Output cost was about $7.10, the largest share of the total. Plotly also notes that K3.1 Pro's OpenRouter input price was about 8 times Kimi K3's ($2.00 versus $0.24 per million tokens), while the output prices were close ($10.00 versus $10.80 per million tokens).

Conclusions and limitations

The main signal from this independent test is proactive analysis. K3.1 Pro retained K3's performance on basic questions, ambiguous questions, and unanswerable requests, while adding the 5 hidden patterns and 2 data-quality issues that K3 failed to find. Its only lost point came from a very specific firmware time-range classification error, showing that a high score does not mean every filter condition is reliable.

The cost is also clear: on Plotly's fixed question set, K3.1 Pro scored close to the maximum, but produced a large amount of reasoning per question, resulting in $13.47 for 43 questions and 2.5 minutes per question. This is better suited to low-concurrency tasks that prioritize complete analysis and data auditing; for interactive, cost-sensitive, or strictly time-limited queries, 42/43 is not enough to evaluate it by itself.

The original article says Moonshot's Kimi K3.1 Pro Highspeed claims the same capabilities at 2–3 times the speed of the standard version, but Plotly did not test Highspeed. The duration, token, cost, and speed conclusions in this article therefore apply only to standard Pro and cannot be transferred to Highspeed.

The original article also says that a full-weights release was planned for 2026-11-04 and that no license had yet been announced. As of the collection date, this plan cannot be written as weights already being available, nor can it support an inference that this OpenRouter run used public weights.

Reproduction notes

1. Obtain access to Plotly's private PlotlyDataAnalyticsBench, synthetic wind-power warehouse, answer key, and authorization; the public article provides neither the complete questions nor a data download.

2. Fix the standard Kimi K3.1 Pro, OpenRouter provider, model version corresponding to the release date, tool schema, SQL executor, timeout, retry, and stopping rules; do not mix in Highspeed.

3. Record the result for each of the 43 questions, and separately record the six categories: analytics, ambiguous, impossible, statistics traps, data quality, and hidden needles.

4. Save the model-call count, tool-call count, query count, input/output/cached tokens, wall-clock time, and cost for each question; distinguish model wait time from SQL time.

5. For the only incorrect answer, check the time range for the “no firmware period” and confirm whether the 2,775 turbine-days would be misclassified into the other-firmware group.

6. If the private question set cannot be obtained, only conduct a methodological comparison; do not claim to reproduce the original article's 42/43, $13.47, or 2.5 minutes, and do not rewrite the individual prompts from the article, to avoid contaminating later tests.