Kimi K3.1 Pro: PacBench single-file game artifact and scoring

One-sentence takeaway

In PacBench's single-prompt, single-file Pac-Man task, Kimi K3.1 Pro scored 93/100, tying Claude Sonnet 5.5; this Kimi run took about 11 minutes and cost $0.68, with 43k output / 22k thinking, while Sonnet 5.5 took about 14 minutes and cost $1.61. This result describes performance on one specific harness and task only; it does not represent overall coding ability.

Use cases

• Tasks this can help assess: Single-file HTML game generation, artifact completeness, runtime stability, cost, and duration for short Agent coding tasks.

• Tasks this should not be generalized to: Multi-file repository maintenance, long-running Agent collaboration, success rates with other toolchains, or an overall ranking of general coding tasks.

• Model version covered: Kimi K3.1 Pro; the request ID was moonshotai/kimi-k3.1-pro, and the actual recorded value was also moonshotai/kimi-k3.1-pro; the metadata identifies its canonical version as moonshotai/kimi-k3.1-pro-20261023.

• Test environment or client: OpenRouter / Claude Code, running in a Cursor cloud VM (Ubuntu 24.04, Claude Code 2.1.303, Node inside the sandbox); no web tools.

• Reasoning level and parameters: high; CLAUDE_CODE_MAX_OUTPUT_TOKENS=128000. OpenRouter was pinned to the Moonshot AI standard endpoint moonshotai/mxfp4, with fallback disabled.

Evaluation method

The PacBench page summarizes the task as a single prompt: Create a Pac-Man game in a single html page. The repository's complete fair-cell prompt requires the model to create the only pacman.html in the current directory, with all HTML, CSS, JavaScript, graphics, and sound embedded. It cannot use external scripts, assets, network requests, or a build step, and must implement the classic maze, four ghosts, power pellets, pause, score, lives, levels, and other features.

This was a single-prompt, single-run fair-cell execution. Kimi ran in an empty sandbox git cell; its 17 response turns included 1 Write, 5 Edits, and 10 Bash commands, and it exited normally. The metadata records the complete fair-cell prompt as its input, with the prompt SHA-256 prefix 55c6d2ec. In the Node canvas stub, the model checked movement, turning, frightened mode, eyes returning home, death, a complete level clear, level-up, and fruit. It attempted to use headless Chrome three times, but the VM blocked it because of socket() permissions.

The score is not a simple function of code length. The README says the baseline retest includes an approximately 90-second automated playtest plus source and maze audits; the checks are Controls 20, Ghosts 25, Pac-Man stuck 20, Maze 20, and Sound 15. The source audit checks the runtime maze grid, reachability, dead ends, ghost house, tunnels, and the real game loop at different frame rates.

The scoring dates need to be separated from the model run date. The README describes the baseline scoring protocol as a retest of Opus 5.5 on 2026-09-28, while v2.md explicitly lists Kimi K3.1 Pro as a later entry added on 2026-10-23. On 2026-10-13, v3 returned 2 points each for Controls and Sound marked minor when there was no major flag; v4, also dated that day, added visual accuracy and arcade easter eggs. The visual portion of v4 used anonymized screenshots, scored separately by two reviewers who did not know the model names. The overall process was therefore not end-to-end blind: the source audit and runtime tests still exposed entry information.

Key results

The unrounded metadata values are: Kimi duration_ms=689066, cost_usd=0.6771788, output_tokens=42703, thinking_tokens=21997; Sonnet 5.5 has duration_ms=865983, cost_usd=1.6051676, output_tokens=123562, thinking_tokens=93156, html_bytes=26587. The leaderboard's displayed 11m, $0.68, 43k, 22k and 14m, $1.61 are card-formatted versions of these metadata values.

Raw data

Kimi's detailed scoring was as follows:

• Controls: minor. Supports arrow keys, WASD, Enter/Space/click to start, P/Esc to pause, and M to mute, with buffered turning, immediate reversal, and pause on loss of focus; after a mouse start, the Start button retains focus, so pressing Space or Enter during the game clicks it again and restarts the game.

• Ghosts: minor. Has four arcade-style target rules, Pinky/Inky upward-direction corrections, chase waves, Elroy, tunnel slowdown, and eyes returning home; the ghost-house point counter resets each time a life is lost, so Clyde consequently remains in the ghost house for 18%–63% of the time.

• Pac-Man stuck: ok. Uses a fixed 1/120 s timestep and maintains 7.64 tiles/s at 30–240 fps; the automated bot cleared 244/244 dots at every frame rate.

• Maze: ok. A classic 28×31 maze matching the arcade layout; 0 dead ends, all 244 dots reachable, left-right symmetric, and Pac-Man cannot enter the ghost house.

• Sound: minor. Has a master bus, continuously rising siren, frightened variants, complete death sound effects, and mute; only 21 of approximately 31 notes in the opening melody are present, there is no bass, and the siren continues playing in a hidden tab.

V4 recorded a visual deduction of −7 and an arcade-easter-egg bonus of +7 for Kimi, so its v3 score of 93 remained 93 under v4. The public record provides no further figures that could support an inference about an overall coding ranking.

Conclusions and limitations

Kimi K3.1 Pro achieved the same 93 score as Sonnet 5.5 on this task with a smaller output and lower recorded cost, and passed maze-reachability and different-frame-rate movement checks. Its main defects were concentrated in interaction focus, ghost-house release counting, and incomplete arcade audio, rather than Pac-Man getting stuck or the maze being impossible to clear.

This was not a strict same-day, same-VM cross-model experiment: Kimi's entry ran on a Cursor cloud VM on 2026-10-23, while the Sonnet entry's metadata identifies OpenRouter / Claude Code, and the two entries ran on 2026-10-23 and 2026-09-28 respectively. They can serve as a public comparison for this task, but the result cannot support a claim that Kimi is faster, cheaper, or stronger in general software development.

Reproduction notes

1. Fix the repository and model versions, and run the complete prompt from bench/prompts/pacman.md exactly once in an empty sandbox cell.

2. Use the repository's bench/run_cell.py, bench/add_entry.py, and playtest/scoring scripts to generate the entry, recording the harness, provider, reasoning level, tokens, duration, cost, and HTML byte count.

3. Open the Kimi playable entry, then compare it with the original fields for slug=kimi-k3.1-pro in entries/meta.json.

4. Use bench/scoring/v2.md, v3.md, and v4.md to distinguish the baseline retest, minor adjustments, and visual/easter-egg adjustments.

This article reviews public artifacts and data; it did not call Kimi again or regenerate the entry. For reproduction, use your own date, model endpoint, and billing records, and do not treat the public values from this run as fixed prices or stable success rates.