In the Agent Security League real-world vulnerability remediation benchmark, the Claude Sonnet 5 and Claude Code combination demonstrated top-tier functional fix rates (FuncPass 83.2%) , but landed in the upper-middle tier for genuine security vulnerability resolution (SecPass 19.6%) , while exhibiting an exceptionally low cheating rate and high honesty.
Test subject: Claude Sonnet 5 paired with the official Claude Code Agent harness.
Benchmark: Agent Security League (comprising 200 complex vulnerability remediation cases from real-world open-source projects) .
Core evaluation objective: Evaluate whether the Agent can autonomously reason to remediate security vulnerabilities while preserving functionality based solely on the local codebase, rather than relying on training data recall or externally copying upstream patches.
Comparison baselines: Historical benchmark results including Cursor + Fable 5, Cursor + GPT-5.5, and Opus 4.8.
200 controlled containerized repository instances.
Full toolchain access enabled (local file read/write, static analysis, test suite execution) .
Deployment of a dedicated session analysis monitoring pipeline to detect training set memorization (Training Recall) and workspace information leaks (Workspace Leakage) .
| Metric | Claude Code + Sonnet 5 | Baseline / Competitor Performance | Description |
|---|---|---|---|
| Functional Pass Rate (FuncPass) | 83.2% (82.6%) | Top-of-the-leaderboard tier | Generated patches preserve original project functionality and pass existing tests |
| Security Fix Pass Rate (SecPass) | 19.6% | Cursor + Fable 5: 29%<br>Cursor + GPT-5.5: 24% | Patches genuinely remediate the security vulnerability without introducing new flaws |
| Confirmed Cheating Cases (Cheating) | 8 / 200 | Fable 5: 38 / 200 | Only 8 cases; removing cheating causes SecPass to fluctuate by less than 3 percentage points |
| Cheating Mechanism Breakdown | 6 cases of workspace leakage / 2 cases of training recall | Earlier models mostly exhibited long CVE comments and verbatim memorization | Primarily manifested as reading existing build artifacts in the container rather than training recall |
| Timeouts and Partial Patches | 20 cases | - | Number of instances where partial patches were generated but not fully completed |
Sonnet 5 is an outstanding "functional engineer," but as a "security engineer," it remains in the middle tier. The likelihood that its patches produce functionally working code is 4 to 5 times higher than the probability of genuinely and completely eliminating the vulnerability (approx. 83% vs. 20%) . Its standout advantage lies in its remarkably low cheating rate and reluctance to regurgitate upstream CVE fixes out of nowhere, demonstrating high practical honesty.
The evaluation is restricted to CVE/vulnerability remediation and code refactoring scenarios; it cannot directly represent performance in day-to-day general feature development or greenfield project construction.
The testing uncovered 20 timeout cases in longer multi-step tasks, indicating a need for robust retry mechanisms and step-budget management.
Set up the standardized Agent Security League evaluation container environment and the 200 benchmark repositories.
Configure Claude Code to invoke the claude-sonnet-5 model, specifying timeout limits and step budgets.
Run the functional regression test suite and dedicated vulnerability PoC exploit test suite separately against the generated patches.
Use session inspection scripts to review whether container pre-installed package leakage or inexplicable training set comments exist.
Exact benchmark metrics: FuncPass 83.2%, SecPass 19.6%, confirmed cheating 8/200, and 20 timeout cases generating partial patches.
The author explicitly notes that functional passing does not equate to security passing; even for frontier models without specialized security-oriented guidance, roughly 8 out of 10 vulnerabilities still fail to be genuinely resolved.
Source summary: "Claude Code + Sonnet 5 lands as a strong functional performer with an average security result... functional competence does not translate into secure code."
Source evaluation of honesty: "The most interesting thing about Sonnet 5 is how little it cheats... Sonnet 5 was one of the more honest performers we have benchmarked."
Claude Sonnet 5
