Kimi K3.1 Pro: Serafim Batzoglou's views on the induction benchmark and Token costs

One-sentence takeaway

Based on a screenshot of Challenge100, the author says Kimi K3.1 Pro tied Claude Opus 5 at 21/100, led Qwen 3.8 Max and Muse Spark 1.3, and trailed GPT-5.6 Sol, GPT-6.1 Sol, and Claude Opus 5.5; he also believes Kimi's Token usage makes it more expensive than GPT-6.1 Sol. The post and GitHub pages provide the leaderboard and some protocol details, but they do not publish per-question raw outputs, a complete reproducible experiment package, or a price bill. The information therefore cannot be reclassified as an independently reproduced benchmark by the author, nor treated as a guaranteed ranking.

Use cases

• Tasks this can help assess: Kimi K3.1 Pro's position in this snapshot of the Challenge100 induction task, and how Token volume, limits, and failure billing affect cross-model comparisons.

• Tasks this should not be generalized to: Fixed rankings for general reasoning, coding, Agent, or all production tasks; or a definitive per-call model cost based on this information.

• Model version covered: Kimi K3.1 Pro; the model ID recorded on the GitHub results page is kimi-k3.1-pro.

• Test environment or client: The GitHub results page records direct calls to Moonshot kimi-k3.1-pro; the client, complete request payload, and service region were not disclosed.

• Reasoning levels and parameters: reasoning_effort=max, max_completion_tokens=500000 (including thinking); other sampling and complete request parameters were not specified.

Evaluation method

The X post shows a Challenge100 table and ranks models by the holdout score from an induction benchmark. The body of the post does not explain the questions, prompts, scoring script, number of requests, or pricing basis; it only attaches a GitHub link.

The linked leaderboard.md explains that Challenge100 is the ordered union of the frozen Challenge64 and mutually exclusive New36; holdout uses five worlds per question; missing responses, vendor errors, empty responses, unfinished outputs that hit the output limit, and invalid parses all count as errors. The table is sorted by Holdout Correct %, Train Correct, Evaluable coverage, and model name.

The linked Kimi results page further states: there were 38 transport retries beyond the 100 original requests; there were 138 physical requests in total, consisting of 100 confirmed terminal states and 38 interrupted original requests whose remote completion status was unknown. Ten pilot tasks used seed 20261023, and the remaining tasks were the exact 90-question complement; there were no tools, prior responses, sampling coverage, or automatic HTTP retries.

These are the linked pages' self-reported details and counting conventions. This article did not independently run the requests, audit remote records, or recheck every question. It therefore cannot claim that the author completed a publicly reproducible independent benchmark.

Key results

The Challenge100-related rows in the X screenshot are below. Output Token units are millions; —* means the screenshot did not report a value.

According to this table, Kimi K3.1 Pro and Claude Opus 5 have the same holdout score. Kimi's Train Correct is higher, but its Evaluable score is only slightly higher; this does not support saying that Kimi outperformed Opus 5 on every task. Kimi is the highest-scoring open-weight model listed in the screenshot, which is an observation within this table and does not amount to a general ranking of all open-weight models.

Raw data

The main post's central judgment is that Kimi K3.1 Pro tied or exceeded Opus 5 on the induction benchmark and clearly led Qwen 3.8 Max and Muse Spark 1.3, but remained clearly below GPT-5.6 Sol and even further below GPT-6.1 Sol and Opus 5.5. It therefore had not reached the frontier, and, based on its Token usage, was more expensive than GPT-6.1 Sol.

In the visible replies, AI Quanting noted that Astra ranked first at 2.0M output Tokens while Opus 5.5 ranked third at 50.6M, and asked whether Challenge100 had a Token limit. The author replied that Astra (OpenAI overall) was limited to 128k Tokens, while Kimi was limited to 1M; most models would exhaust their Tokens on these questions, OpenAI did not charge for failures, and Anthropic charged for failures, so failed Tokens were included in the total.

The 128k-versus-1M figures are the author's limit terminology in an X reply. The post does not define whether they refer to context, output, or a total server-side budget. The GitHub Kimi results page separately records max_completion_tokens=500000; the two figures must not be conflated as the same parameter.

The Kimi results page gives the following known usage: 594,862 input tokens and 11,539,644 completion tokens, for a total of 12,134,506; completion already includes 11,530,392 reasoning tokens and 9,252 visible tokens, so reasoning must not be added again. There are also 79,616 cached input Tokens, but the cache usage for 76 requests is unknown; the input and completion usage for the 38 interrupted original requests is unknown. The page explicitly says “pricing and billing are unknown,” so these figures cannot be treated as zero cost or a billing estimate.

GitHub's Token JSON records Kimi's output_tokens as 11,539,644, physical_attempts as 138, and the unreported response lower bound as 38. The screenshot's Token amounts for other models are likewise output amounts and cannot be directly converted into costs: prices, cache discounts, failed retries, plan allowances, and vendor settlement rules were not disclosed in a consistent way.

Conclusions and limitations

• This X post can serve as a community-opinion snapshot dated 2026-10-24: in the Challenge100 table shown in the screenshot, Kimi K3.1 Pro scored 21/100, tied with Opus 5, and scored below GPT-5.6 Sol, GPT-6.1 Sol, and Opus 5.5.

• Its 11.5M output Tokens exceed Astra's 2.0M, GPT-6.1 Sol's 4.1M, and Muse Spark 1.3's 4.7M, but are below Opus 5.5's 50.6M and Opus 5's 27.1M; Token volume alone cannot determine cost or quality.

• The author's statement that Kimi was “more expensive than GPT-6.1 Sol” is an opinion, not a public billing comparison. The GitHub page explicitly marks Kimi's pricing and billing as unknown.

• The 128k/1M claim about failure billing comes from the author's reply and is not supported by vendor billing or request-level evidence. Kimi had 38 interrupted original requests whose remote completion status was unknown; one cannot assume that they were canceled, unbilled, or necessarily billed.

• The GitHub page says “completed, independently audited campaign,” but the public page does not include per-question raw responses, the complete prompt, remote request logs, or audit materials. This article only reports that self-description; it does not upgrade the leaderboard into an author-independent benchmark verified by this article, nor treat the ranking as guaranteed.

• Challenge100's five holdout worlds, question composition, retry behavior, and counting rules affect the result. Models also differ in limits, failure handling, and the range of reportable Tokens, so output volume alone cannot support a cross-vendor value-for-money ranking.

Reproduction notes

To reproduce this claim, fix the versions of Challenge100's Challenge64/New36, the five holdout worlds, the original prompt, JSON contract, model ID, reasoning_effort, completion cap, timeout, concurrency, retry policy, and selection rules. For each request, save the input, output, whether thinking is included, terminal state, failure reason, and billing status.

Costs should be calculated under a consistent basis:

Total cost = uncached input Tokens × input price per Token
           + output Tokens × output price per Token
           + cache-hit input Tokens × cached input price per Token
           + confirmed charges for failed/retried requests

Unit prices should be normalized to a per-Token price; if the vendor also charges for cache writes, include that charge to avoid double-charging cached input.

For the 38 interrupted requests whose remote completion status is unknown, record “unknown” separately rather than filling it in as 0. Only after obtaining request-level vendor billing, complete raw outputs, and benchmark artifacts that can be independently checked should one discuss cost rankings or stability across repeated runs.