GPT-5.6 Luna Semgrep IDOR Security Benchmark and Cost per True Positive

One-sentence takeaway

Semgrep's security evaluation found that Luna costs roughly 6 times less per true positive than heavier models, with only a marginal F1 sacrifice; however, precision and recall are strongly dependent on the harness, so raw-model scores must not be treated as production protection capability.

Test environment

• Models: GPT-5.6 Luna, Terra, Sol, and GPT-5.5 as the control.

• Task: Primarily IDOR detection, with mentions of vulnerability classes such as SSRF; the tests used representative production applications.

• Repetitions: The article says that security researchers rigorously reviewed the true positives for each benchmark and ran it ten times.

• Two conditions: The model on its own with a guided prompt, and a harness equipped with Semgrep tools, scaffolding, and a complete security workflow.

• Metrics: recall (finding true positives), precision (the share of reported findings that are real vulnerabilities), and the F1 harmonic mean; cost per true positive was also counted.

Input/configuration

The article did not publish complete IDOR samples, prompts, model snapshots, per-run outputs, pricing bills, or the CSV underlying the charts; the page body provides the methodology, directional results, and cost multiples. The production applications, researcher review, and ten runs are clues for reproducing the method, but are insufficient for recomputing it item by item.

Results data

• All GPT-5.6 reasoning levels outperformed GPT-5.5 on this IDOR benchmark.

• Compared with GPT-5.5, GPT-5.6 improved true-positive recall by more than 2 times, but precision regressed, meaning more false positives.

• Luna's cost per true positive is the standout result: the article says it is about 6 times cheaper than heavier models, while giving up only a marginal amount of F1.

• The article presents OpenAI CTF's 96.7% as a "result with a harness," not as raw-model capability; after removing the preconfigured prompt and tool environment, one cannot assume that the figure would remain the same.

• The article's IDOR charts are presented as images; the readable page text does not provide exact recall, precision, or F1 values for Luna, Terra, and Sol, so no unverified numbers from the charts are added here.

Conclusions

• For high-volume, narrow-scope vulnerability triage that can be reviewed by security researchers, Luna may be a cost-effective first-pass filter.

• The cost of high recall is lower precision; production systems should use static analysis, reachability checks, manual review, or a second model to filter false positives.

• Tools and the harness are not ancillary: Semgrep's results clearly support the view that the combination of "model + tools + scaffolding" determines the effectiveness of the security workflow.

Limitations

• Semgrep is a security product vendor, so the test tasks, toolchain, and commercial objectives may affect the results; this should be viewed as an independent engineering evaluation rather than a blind test.

• The article discloses only directional multiples, not per-sample data, complete prompts, or the raw chart values, so the 6-times figure or the F1 difference cannot be independently recomputed.

• IDOR and production-application security tests do not represent malware, supply-chain, compliance-audit, or other vulnerability categories.

• CTFs and other public benchmarks may have training-data contamination or harness bias; the article itself also warns that models and harnesses must be evaluated together.

Reproduction steps

1. Prepare de-identified representative applications and IDOR/SSRF tasks, first freezing the vulnerability labels and the security researchers' review rules.

2. Fix Luna's model alias, reasoning effort, guided prompt, tool permissions, and timeout; run it at least ten times and save the complete outputs.

3. Run GPT-5.5, Terra, or Sol as controls under the same conditions; then separately enable and disable the security-tool harness.

4. For each run, calculate recall, precision, F1, false-positive count, manual review time, input/output tokens, and dollar cost.

5. Report cost per true positive, and put the "raw model" and "complete security workflow" in separate tables to avoid attributing the harness gain to Luna.

Source excerpt or observation (for compliance short quote only)

Semgrep's core judgment is “Luna as a cost per true positive was a standout result,” but the same article emphasizes that the harness can significantly change the result.