In a scoped cyber assessment it beat GLM-5.2, but ACE was 0/41

Test environment

• Evaluation target: Moonshot AI Kimi K3; the focus is cyber capability, not general code quality.

• ExploitBench: 41 recent vulnerability tasks related to the V8 engine and JavaScript/WebAssembly, measuring progress from coverage/crash reproduction to arbitrary code execution (ACE).

• The Last Ones (TLO): A simulated enterprise-network attack path with 32 steps, 4 subnets, and roughly 20 hosts; the standard limit is 100M tokens.

• To reduce refusals, closed-source U.S. models had system-level safeguards disabled; because of its hosted setup, K3 ran only a selective cyber evaluation.

Input/configuration

• K3’s overall cyber capability is estimated primarily from a single ExploitBench result, so its confidence interval is wider than those of other models.

• The NIST page publishes the benchmark scale, task characteristics, and explanation of the confidence interval, but not K3’s complete system prompt, toolchain, or individual trajectories.

Results

Conclusion

In this constrained evaluation, K3 clearly outperformed GLM-5.2 but still trailed the strongest cyber models, and it did not reach ACE on any of the 41 ExploitBench samples. It can be considered a security-research candidate that requires strict isolation and authorization controls, but should not be described as already having stable end-to-end attack capability.

Limitations

• This is a preliminary assessment with a small sample, and K3 ran only a selective evaluation; the confidence interval for its overall capability estimate is wide.

• TLO had no active defenders or alert penalties and used a preset attack path, so it differs from a real enterprise network.

• Results with safeguards disabled for closed-source models do not represent the behavior users can directly obtain from public products.

• These data answer questions about controlled cyber benchmarks only; they cannot be generalized to all defensive, vulnerability-audit, or production-security tasks.

Reproduction steps

1. Obtain only the public benchmarks and version notes cited by NIST, and do so in an isolated, authorized lab environment.

2. Fix the model version, token limit, tool permissions, and scoring script; start with low-risk vulnerability-identification and remediation tasks.

3. Record discoveries, reproducibility, false positives, tool calls, and stopping reasons; do not extend the security evaluation to unauthorized testing of real systems.

4. Put K3 and GLM-5.2 or other baselines on the same task set, and report confidence intervals rather than a single success rate.

Original evidence and data

The NIST page publishes 41 ExploitBench tasks, 32-step TLO, 32%/24%, ACE 0/41, 17/32, 1/10, and the 100M-token limit, and explicitly explains the differences between the benchmarks and real environments.

Source excerpt or observation (short compliance quote only)

The page’s comparison conclusion is: “Kimi K3 outperforms GLM-5.2”.