Semgrep IDOR Benchmark: GLM-5.2 Results with a Prompt-Only Setup in Security Code Auditing

Test environment

• Task: Detecting IDOR (insecure direct object references), using real open-source applications from a dataset used in earlier research.

• Metrics: Calculate precision, recall, and F1 from known true positives, and record the cost of each true positive.

• Fixed factors: The same IDOR dataset, evaluation method, and system prompt.

• Variables: The model and the harness. GLM-5.2, MiniMax M3, and Kimi K2.7 Code received only the prompt and codebase in a simple Pydantic AI harness; Claude Code used the Claude Code SDK; Semgrep Multimodal used a separate dedicated harness for endpoint enumeration and targeted context.

Inputs / configuration

• Model: GLM-5.2; Pydantic AI; a unified IDOR system prompt.

• Publicly described prompting strategy: Provide a search strategy and the key points for identifying IDOR, but no endpoint-discovery scaffolding.

• Run scope: One dataset / one experiment; the authors explicitly state that this is a harness comparison experiment, not a general model leaderboard.

Results

• The cost for GLM-5.2 to find each true positive was approximately $0.17.

• GLM-5.2's F1 was 7 percentage points higher than Claude Code's 32%, but it was still below Semgrep's dedicated pipeline with endpoint discovery.

Conclusion

With the same minimal prompt and a simple harness, GLM-5.2 achieved a higher F1 on IDOR detection than the Claude Code configuration tested here, reaching a usable result at lower cost. The larger gap came from whether the harness provided endpoint enumeration and targeted context, so model selection cannot be separated from the workflow scaffolding.

Limitations

• One vulnerability category, one limited dataset, and one run; the authors explicitly note that other vulnerability types, such as SSRF, may produce a different ranking.

• GLM-5.2 and Semgrep Multimodal did not use the same harness, so 39% and 61% cannot be treated as a pure model difference.

• The blog does not publish the complete dataset, per-sample predictions, random seed, or full prompts; it is suitable for reviewing the method, but does not constitute a fully reproducible out-of-the-box setup.

Reproduction steps

1. Obtain the same public IDOR dataset, and fix Pydantic AI, the model SDK, timeouts, retries, and output parsing.

2. Use the unified system prompt to run GLM-5.2 and the other models separately, retaining every vulnerability judgment and cost log.

3. Calculate precision, recall, and F1 from the known true positives; then add an endpoint-enumeration harness to separate the model contribution from the scaffolding contribution.

4. Re-test on a second vulnerability category and across multiple random runs; do not generalize a one-off result into a general security ranking.

Original evidence and data

• The Semgrep article reports a 39% F1 for GLM-5.2, 32% for Claude Code (in the body text), and a cost of approximately $0.17 per true positive.

• The original also lists MiniMax M3 at 23%, Kimi K2.7 Code at 22%, GPT-5.5 Codex at 20%, and DeepSeek V4 at 17%, and explicitly states that the open-source models did not receive endpoint-discovery scaffolding.

Source excerpt or observation (compliant short quotation only)

• The original frames the question as “how much do model capabilities and harness capabilities each contribute,” which is more useful for guiding agent design than a single leaderboard.