Kimi K3 code security evaluation: strong benchmarks do not guarantee precision

Original article

Open weight and open source models are having a field day right now. They're smashing benchmark after benchmark, making waves on social media, and are now the subject of proposed US bans framed as cyber security measures. If you lead a security organization, you're probably being asked by your board, your engineers, your procurement team, or really anyone who's looking to curb token use, whether these models are ready to do real security work on your infrastructure.

And in a previous post we saw the first real contender: GLM. It was the first open weight model we evaluated that genuinely impressed us. Shortly after, we saw OpenAI's answer to Anthropic's Mythos/Fable line with the release of Sol, which we also benchmarked (and were surprised at how cheaply the Luna model could find valid true positive vulnerabilities). So when Kimi K3 was released late last week, we were ready to watch another open weight model shoot past the expensive frontier models.

It didn't.

Yes, we've seen the cyber benchmarks. We've seen the coding benchmarks. It's good — but it's not that good. We ran the numbers again to make sure. We delayed this post, again to be sure. We're sure now.

What is an Open Weight Model?

If you're familiar with AI, you probably know the frontier model families: Anthropic's Claude, OpenAI's GPT, and the individual model versions within them, like Claude Fable or GPT 5.6.

An open weight model is still a large language model, but instead of accessing it through someone else's cloud, you download it and run it on your own infrastructure. That's a significant advantage for a security organization: full control over where your code goes.

An open source model goes further, you get the model and all the code used to create it. With an open weight model, you have the weights, so you can audit some of what the model does, but you never see the original training code.

A harness is the prompts, UI, and tooling wrapped around a model. In theory, any harness works with any model; in practice, it's rarely that simple. Harnesses can be open source too - we recently compared several.

Distillation is the process of training a smaller (or cheaper-to-run) model to imitate the outputs of a larger "teacher" model. You generate large volumes of responses from the teacher and train the student to reproduce them. The student inherits much of the teacher's behavior at a fraction of the inference cost. This is what many open weight models have been accused of doing to frontier models.

What Are We Testing For?

We got a lot of comments after the first blog about how we test models, so here’s a brief recap: we benchmark models on their ability to detect Insecure Direct Object Reference (IDOR) vulnerabilities in real, open source codebases, scoring four things:

Precision — what fraction of detected IDORs are real?

Recall — what fraction of real IDORs were detected?

F1 — the harmonic mean of precision and recall

But why just IDORs, why not tell it to find every vulnerability? A lot of cyber benchmarks do this but when you task a bare model with a full security sweep, results vary wildly between runs, which makes it difficult to draw statistical conclusions or generalize across repositories. And because we’re not necessarily controlling for harness (as our benchmarks were born out of testing our harness against frontier harnesses) focusing on a single vulnerability class lets us measure how deep a model can go rather than how broad, make sound statistical interpretations, and build a real understanding of how these models behave. That is to say that we don’t necessarily need to test every vulnerability. While these results may not entirely generalize to other vulnerability classes, they give us a holistic view of the model’s cyber capabilities.

For each model, we provide a prompt specific to the vulnerability class under test. For IDORs, that prompt includes: what an IDOR is, what to look for, what not to look for, a suggested investigation strategy, reporting rules, and an output format. A typical benchmark run works like this:

Clone the benchmark repo (an experiment covers multiple repos)

Invoke the model with our prompt

Attach a minimal agentic harness (we use pydantic_ai) so the model can traverse the repo

Let the agent explore and find vulnerabilities, completely isolated from researchers

The agent returns results, and our internal benchmarking system scores the experiment with a set of deterministic scoring engines

The agent completes the benchmark

The goal is to give the model only what it needs to act as an agent, and let it work uninfluenced beyond the initial prompt. We repeat this across multiple repos and multiple parameter combinations, effort levels, harnesses, tools, and plugins so we have a good overall understanding of its performance. We also attempt to identify cheating by the model on benchmarks, but you can learn more about that in our other blog.

When we’re ready to validate we use determini stic, rule-based scoring engines throughout. This keeps experiments reproducible and scores every model identically. We also run a separate set of internal "grounding" benchmarks to assess how models generalize outside our testing environments. We’re looking for a lot of different things here. A higher-precision model gives you more accurate classifications but will likely miss the more complex vulnerabilities hiding in your codebase. A higher-recall model casts a wider net and finds more (at the cost of more false positives). And F1 balances the two and is what we typically use for comparison. While that lets us rank models from “worst” to “best,” it’s important to recognize that different organizations, security tasks and goals can lead to a different model being the right choice. 

Our Results

Configuration

Harness

Precision

Recall

F1

GPT-5.6 Sol

Guided prompt

0.880

0.250

0.389

GLM-5.2

Guided prompt

0.863

0.238

0.345

Claude Opus 4.8

Guided prompt

0.910

0.210

0.343

Kimi K3

Guided prompt

0.684

0.226

0.340

GPT-5.6 Terra

Guided prompt

0.870

0.200

0.330

GPT-5.6 Luna

Guided prompt

0.860

0.190

0.314

GPT-5.5

Guided prompt

0.840

0.140

0.238

At first glance, Kimi K3 looks fine. Its guided-prompt F1 of 0.340 is within noise of GLM-5.2 (0.345) and Claude Opus 4.8 (0.343). This is why it’s important not to just measure F1 and call it a day.f we stopped at the aggregate, this would be a very different blog post. Because here’s the thing, two things stood out to us immediately

First, the precision gap. Every other guided-prompt configuration lands between 0.84 and 0.91 precision. Kimi K3 lands at 0.684. In practice, that means a meaningfully larger share of Kimi's findings are false positives that your team must investigate and dismiss. Even if inference is inexpensive, Kimi shifts more of the total cost to human triage.

Second, and more importantly, repo scale. Kimi K3 averaged roughly 6% F1 on Repo D, the largest enterprise-style repository in our benchmark set. We ran several additional experiments on that repository alone, and the underperformance persisted. GLM and the frontier models averaged around 20% F1 generation and other open-weight models were less grounded than frontier models. We have not yet run K3 through that grounding audit, and one fixture cannot prove that repository size alone caused the result. Taken together, however, the evidence points toward K3 having difficulty maintaining grounded security reasoning as codebases become larger and more interconnected.

We cannot run frontier labs' proprietary harnesses (Claude Code security-review and the like) on open weight models, so our cleanest head-to-head remains the guided prompt, where every model gets exactly the same treatment. What we can do is adapt our own harness, and we did, running Kimi K3 through the same Semgrep Multimodal harness as the frontier models. But Kimi still ends up at the bottom.

Configuration

Harness

Precision

Recall

F1

GPT-5.6 Sol

Semgrep Multimodal

0.530

0.730

0.617

GPT-5.6 Luna

Semgrep Multimodal

0.580

0.650

0.616

GPT-5.6 Terra

Semgrep Multimodal

0.560

0.630

0.593

GPT-5.2

Semgrep Multimodal

0.630

0.540

0.581

GPT-5.5

Semgrep Multimodal

0.730

0.360

0.481

Claude Opus 4.8

Semgrep Multimodal

0.730

0.320

0.448

Kimi K3

Semgrep Multimodal

0.595

0.301

0.400

Kimi K3 may be a reasonable option for smaller, more self-contained projects or teams willing to absorb a higher triage burden, but our results suggest it is not a drop-in replacement for GLM or frontier models on large, interconnected codebases. Teams evaluating it for those environments should test it against representative repositories and expect to need stronger scaffolding to narrow and validate its analysis.

Self-Hosted Models Make a Splash

There are real, legitimate reasons security organizations care about open weight models:

Data residency: In many regions, sending source code to a US cloud provider is a compliance problem, not a preference. A self-hosted model keeps your code on your infrastructure.

Auditability: You can inspect an open weight model, while the tools are limited this is theoretically possible with open weight models.

Cost: These models simply cost less than frontier models, for a similar performance on some task

Lack of Guardrails: This is likely partly why these models are so good at cyber benchmarks, lacking a lot of the guardrails put in place by Frontier Labs.

That last one is kind of  a problem though, and might lead to restrictions on open weight models in the US due to fears of cyber attacks and potential poisoning. Over the past year, various options have been suggested, such as adding Chinese AI labs to the Commerce Department 's Entity List, a joint NSA and Office of the National Cyber Director advisory on Chinese AI lab threats, an executive order requiring US companies hosting Chinese models to guarantee their security and accept liability for breaches, and draft supply chain rules targeting Chinese open source models. Many have criticized these proposals as motivated more by fear of competition from open source models than legitimate security concerns.

Through our testing we found no evidence of security backdoors in open weight models. The larger risk for security teams may be choosing the wrong model for the wrong task, slowing down and adding friction to teams. If you're evaluating open weight models for security work, the benchmark headlines are not enough. Our recommendations:

Test on repos that look like yours. Kimi K3's aggregate numbers looked competitive; its performance on enterprise-scale repos did not. If your codebase is large, benchmark on large codebases.

Weigh precision against your triage capacity. A model that finds more but is wrong more often shifts cost onto your team.

Treat GLM and Kimi K3 differently. In our testing, GLM behaves like a viable Opus alternative. Kimi K3 does not, at least not without a serious harness around it, keep this in mind when you choose models for tasks.

Audit what you self-host. The ability to audit is the single biggest security advantage of open weight models. Use it.

In the meantime we'll continue benchmarking new models as they're released, open weight, open source, frontier, distilled and everything in between.

Lots of love,

Semgrep Security Research and Engineering