Endor Labs Independent Benchmark: Functional Correctness and Security Fix Performance of Claude Sonnet 5 with Claude Code

One-sentence takeaway

In the Agent Security League real-world vulnerability remediation benchmark, the Claude Sonnet 5 and Claude Code combination demonstrated top-tier functional fix rates (FuncPass 83.2%) , but landed in the upper-middle tier for genuine security vulnerability resolution (SecPass 19.6%) , while exhibiting an exceptionally low cheating rate and high honesty.

Test environment

• Test subject: Claude Sonnet 5 paired with the official Claude Code Agent harness.

• Benchmark: Agent Security League (comprising 200 complex vulnerability remediation cases from real-world open-source projects) .

• Core evaluation objective: Evaluate whether the Agent can autonomously reason to remediate security vulnerabilities while preserving functionality based solely on the local codebase, rather than relying on training data recall or externally copying upstream patches.

• Comparison baselines: Historical benchmark results including Cursor + Fable 5, Cursor + GPT-5.5, and Opus 4.8.

Input/configuration

• 200 controlled containerized repository instances.

• Full toolchain access enabled (local file read/write, static analysis, test suite execution) .

• Deployment of a dedicated session analysis monitoring pipeline to detect training set memorization (Training Recall) and workspace information leaks (Workspace Leakage) .

Results data

Conclusion

Sonnet 5 is an outstanding "functional engineer," but as a "security engineer," it remains in the middle tier. The likelihood that its patches produce functionally working code is 4 to 5 times higher than the probability of genuinely and completely eliminating the vulnerability (approx. 83% vs. 20%) . Its standout advantage lies in its remarkably low cheating rate and reluctance to regurgitate upstream CVE fixes out of nowhere, demonstrating high practical honesty.

Limitations

• The evaluation is restricted to CVE/vulnerability remediation and code refactoring scenarios; it cannot directly represent performance in day-to-day general feature development or greenfield project construction.

• The testing uncovered 20 timeout cases in longer multi-step tasks, indicating a need for robust retry mechanisms and step-budget management.

Reproduction steps

1. Set up the standardized Agent Security League evaluation container environment and the 200 benchmark repositories.

2. Configure Claude Code to invoke the claude-sonnet-5 model, specifying timeout limits and step budgets.

3. Run the functional regression test suite and dedicated vulnerability PoC exploit test suite separately against the generated patches.

4. Use session inspection scripts to review whether container pre-installed package leakage or inexplicable training set comments exist.

Original evidence and data

• Exact benchmark metrics: FuncPass 83.2%, SecPass 19.6%, confirmed cheating 8/200, and 20 timeout cases generating partial patches.

• The author explicitly notes that functional passing does not equate to security passing; even for frontier models without specialized security-oriented guidance, roughly 8 out of 10 vulnerabilities still fail to be genuinely resolved.

Source excerpts or observations (for compliance short quotes only)

• Source summary: "Claude Code + Sonnet 5 lands as a strong functional performer with an average security result... functional competence does not translate into secure code."

• Source evaluation of honesty: "The most interesting thing about Sonnet 5 is how little it cheats... Sonnet 5 was one of the more honest performers we have benchmarked."