Paweł Huryn: Comparing Kimi K3.1 Pro on bug injection across two repositories

One-sentence takeaway

The author planted 105 reproducible bugs in two real repositories and asked the model to find and fix them; in the set of results he published, Kimi K3.1 Pro max scored 36/105, below DeepSeek V4.1 Flash 1017 max at 42.7, Muse Wonder Full 1.0 max at 42, and Opus 5.5 max at 41.7. Kimi's actual billed cost was $13.76, which led the author to conclude that its cost was not attractive.

Use cases

• Tasks this can help assess: Locating and fixing bugs in existing code repositories, along with Agent coding efficiency; comparing repair scores, runtime, and per-run cost under a fixed harness.

• Tasks this should not be generalized to: General programming ability, the ability to discover defects that were not planted, frontend visual quality, all repository types, or absolute rankings across different harnesses.

• Model versions covered: Kimi K3.1 Pro; comparators included DeepSeek V4.1 Flash 1017, Muse Wonder Full 1.0, Opus 5.5, Muse Spark 1.3, and Sonnet 5.5 in the author's later replies.

• Test environment or client: Bug Hunt Bench; the coding benchmark used the Claude Code harness. The author said the Kimi launch post also used Claude Code, while other benchmarks used other harnesses.

• Reasoning levels and parameters: The main result is labeled max; the author also published max, xhigh, high, medium, and low results for Opus and Sonnet. The complete system prompt, task list, repository names, sampling parameters, and per-task records were not disclosed.

Evaluation method

1. Plant 105 reproducible bugs in two real repositories.

2. Ask the model to independently find and fix the problems it can identify in an Agent coding environment.

3. Score the number of verified planted bugs fixed; fixes for defects that were not planted do not count.

4. Compare repair scores, wall-clock runtime, and cost. The author noted in the screenshot that the leaderboard shows 24 selected runs out of 128 runs, sorted by the number of fixes.

The author did not publish a per-task list for the two repositories and 105 bugs, or complete run logs. This record therefore covers the visible aggregate results rather than a fully reproducible experiment package.

Key results

The aggregate scores in the main post were:

In a later reply, the author also listed three scores for the same Kimi max configuration: 36, 42, and 30. This indicates that repeated runs were variable. The main-post score and individual runs should be recorded separately; the author did not provide the full context, order, or synchronized conditions for each run or for the other models.

In the same reply, the author listed the following Opus 5.5 scores: max 43/39/43, xhigh 35/37/36, high 32/32/31, medium 30/30/31, and low 23/25/19. For Sonnet 5.5, the scores were: max 57/54/43, xhigh 39/34/35, high 32/34/37, medium 18/28/24, and low 20/17/20. The author specifically noted that the first Sonnet max run completed seven times as many rounds as Astra, calling it the most stubborn combination.

The author also said that Kimi max was worse than DeepSeek V4.1 Flash 1017 at high effort. He would normally continue testing, but chose to stop this time to avoid consuming more tokens.

Raw data

The Bug Hunt Bench leaderboard in the main post's attached image was marked as updated on 2026-10-23. It listed scores counting only planted bugs, uncounted defects that were not planted, wall-clock minutes, and dollar cost. The rows directly related to this post were:

The note at the bottom of the screenshot says that the costs for DeepSeek V4.1 Flash 1017, Kimi K3.1 Pro, and MiMo-V2.6-Pro are actual bills. Other costs are list-price estimates. The Grok 4.7 figure was reconstructed from the CLI-reported context usage shown below the benchmark and is not a bill. Therefore, $13.76 cannot be directly treated as the same cost measure as Opus's $58.53 or Sonnet's $154.03.

In a reply, the author summarized the cost difference by saying that in past Kimi K3 testing, Moonshot CLI cost about 2.9 times as much as Claude Code. He also explained that he chose Claude Code because the Kimi launch post's coding benchmark used that harness. This 2.9x figure is the author's description of an earlier test and does not constitute an independent cost table for this 105-bug experiment.

The author believes that non-planted bugs should not count: some models report unrelated real “issues” to increase their discovery count, 99% of which are low priority, and fixing them could even harm the product. He said the planted bugs represent the distribution of important problems he had actually handled in the past, and that frontier models had overlooked these problems in early 2026.

Author-visible replies

• The author said he skipped one anomalous run, without explaining whether this affected the aggregate statistics. Its score therefore cannot be recalculated here.

• In response to a question about using the Claude Code harness, the author said that the Kimi K3.1 Pro launch post's coding benchmark used Claude Code; in his earlier Kimi K3 testing, he also observed that Moonshot CLI cost about 2.9 times as much as Claude Code.

• The author agreed that testing should continue at lower effort levels, but considered Kimi max already below DeepSeek V4.1 Flash 1017 high effort and stopped to avoid consuming more tokens.

• The author added that Moonshot's coding benchmark used Claude Code, while other benchmarks used other harnesses; therefore, results from different leaderboards should not be merged directly.

• In the discussion of whether non-planted bugs should count, the author said he could not score them and explained why he counted only planted bugs. The reply also said that he would not spend more effort levels on testing and that he was curious about Kimi's performance on Vulcan Bench.

Conclusions and limitations

• In this two-repository test with 105 planted bugs, Kimi K3.1 Pro's aggregate score in the main post was 36/105, below the main-post results for DeepSeek, Muse Wonder Full, and Opus.

• Kimi's three max results were 36, 42, and 30, showing substantial variation across repeated runs; a single score of 36 cannot be used to infer a fixed success rate.

• The leaderboard screenshot gives both runtime and cost. Kimi max took 64.8 minutes and incurred an actual $13.76 bill, compared with DeepSeek max's 62.1 minutes and actual $1.84 bill; Kimi had a lower repair score and a higher bill.

• The screenshot's costs for Opus and Sonnet are list-price estimates, while Kimi and DeepSeek use actual bills. Cost comparisons can therefore be treated only as observations under the leaderboard's stated accounting.

• The author's comparison depends on the Claude Code harness, and he explicitly said that other benchmarks used other harnesses. The combination of model, CLI, context management, tool loop, and stopping conditions can affect the result.

• The sample covers only two repositories and 105 planted bugs. The complete repositories, per-task inputs, repair patches, failure reasons, and token details for each run were not published; the result cannot be generalized to all codebases or to discovering non-planted defects.

• In the visible replies, the author said he would not continue testing other effort levels and was curious about Kimi's performance on Vulcan Bench. These are future plans or interests, not results from this test.

Reproduction notes

At minimum, reproducing this evaluation would require the author's unpublished snapshots of the two repositories, 105 planted bugs, verification answers, Claude Code harness configuration, model version, permissions and stopping conditions, plus the repair patch, verification result, wall-clock time, and bill saved separately for each run. The public post is sufficient only to check the aggregate scores, leaderboard screenshot, and the author's explanation of the cost accounting; it does not support a claim that the evaluation has been reproduced task by task.

Short source quotes

• “2 repos. 105 replanted bugs. Find and fix what you can.”

• “Kimi K3.1 Pro max: 36, 42, 30”