GPT-6 Sol: KillSwitch-Bench Adversarial Esoteric-Language Coding Agent Benchmark

One-sentence takeaway

The KillSwitch-Bench leaderboard lists GPT-6 Sol (Codex harness) with a composite score of 28.0%, a cost of $0.42 per task, and a runtime of 2m15s. The 28.0% is a composite score based on pass rate, cost, and code size; it must not be read as accuracy or the task pass rate.

Test setup and method

• Task: Coding agents implement or understand KillSwitch programs. KillSwitch is deliberately adversarial: an esoteric programming language whose source code hides the real logic in misleading text.

• Input materials: Each agent receives a task prompt and the KillSwitch language specification. Some tasks also provide a script file for the program to be implemented. The specification repository publishes SPEC.md and AI_SPEC.md.

• Evaluation: Tasks have known answers and are scored deterministically by a program; no LLM judge is used.

• Isolation and limits: Each run uses an isolated container. The per-task budget cap is $5, with a hard timeout of 30 minutes. Agents cannot access the internet and can access model APIs only through a restricted proxy.

• Repeat runs: Each model is run at least twice on every task. The page does not report GPT-6 Sol's actual total number of runs, aggregation method, or variance.

• Harness: The leaderboard evaluates models together with coding agents. GPT-6 Sol uses Codex; rankings are also affected by the harness, tool configuration, and implementation.

• Scoring: The page says the composite score combines pass rate, cost, and code size, rewarding solutions that pass tasks with low cost and short implementations. It does not publish the complete formula or enough component data to recalculate the score.

Inputs and configuration

• The leaderboard's model label is gpt-6-sol, provided by OpenAI; the agent is Codex.

• The leaderboard reports a composite score of 28.0%, a cost of $0.42 per task, and a runtime of 2m15s for GPT-6 Sol. The page does not specify the reasoning level, model snapshot, exact API parameters, or how these values were aggregated.

• The page links to the KillSwitch project repository and language specification, but does not show GPT-6 Sol's task prompts, per-task code, run logs, or failure cases. The repository README points to a benchmark submodule; the currently visible submodule link cannot be opened (it returns 404), so the task list and original run materials cannot be checked against it.

Results

The leaderboard notes, “Score combines pass rate, cost, and code size.” The percent sign in 28.0% is therefore the display format for this composite score; it does not mean GPT-6 Sol passed 28.0% of the tasks. The page does not provide a separate pass rate, so the number of tasks passed cannot be inferred from the composite score.

Comparisons

• The leaderboard also lists GPT-5.6 Sol (Codex) at 27.4%, $0.24 per task, and 3m24s. The comparable figures here are the leaderboard's composite score, cost, and time; the composite score is higher by only 0.6 percentage points, which must not be described as a 0.6-point increase in pass rate.

• GPT-6 Astra (Codex) is listed at 57.8%, $1.45 per task, and 1m37s. It shares the Codex name with Sol, but the public page does not provide enough configuration detail to establish that their run settings were identical.

• Other top-ranked entries use different harnesses, including Claude Code and OpenCode. The leaderboard explicitly tests coding agents and models as combined systems, so rankings across models also reflect differences in agents and toolchains.

Conclusion

• On this adversarial-language task set and with the Codex configuration, GPT-6 Sol has a leaderboard composite score of 28.0%; the page reports a cost of $0.42 per task and a runtime of 2m15s.

• This result describes GPT-6 Sol's composite performance on this specific agent-coding benchmark. It is not general code accuracy and cannot directly represent performance on common programming languages, real-world repository maintenance, or other agent configurations.

• Compared with GPT-5.6 Sol, the composite score is slightly higher, per-task cost is higher, and runtime is shorter. Because component pass rates and per-task results are not published, this does not establish a change in pass rate or a significant improvement in cost-effectiveness.

Limitations and verification

• The page does not state when the leaderboard was published or when the runs took place. This table is a snapshot collected on 2026-09-23; the leaderboard may change later.

• The site publishes only the scoring components—pass rate, cost, and code size—not the complete formula, weights, raw component scores, per-task results, or actual number of repeat runs. The composite score cannot be independently recalculated.

• “At least two runs per task” is the benchmark's minimum repeat-run rule. It does not mean there were only two runs, nor does it establish whether the leaderboard score is an average, best result, or another aggregate.

• The $5 per-task budget and 30-minute timeout are upper limits, not GPT-6 Sol's measured cost and runtime; its reported values are $0.42 and 2m15s, respectively.

• The leaderboard does not show directly accessible links to individual tasks or GPT-6 Sol's code. The KillSwitch repository publishes the language specification; the benchmark submodule link referenced by its README returned 404 when checked for this note.

• To reproduce the result, preserve the leaderboard version, tasks and specifications, model snapshot, Codex/harness version, budget, timeout, and each run's pass status, cost, time, and code size. The complete scoring rules must be obtained from the benchmark provider before recalculating the published score.

Source excerpt or observation (brief quotation for compliance)

The methods page describes the tasks as using “Deterministic evaluation” and says the score combines pass rate, cost, and code size. The GPT-6 Sol result should therefore retain its composite-metric qualification.