SonarSource: Evaluating Claude Opus 5.5 on Java Code Generation

One-sentence takeaway

On SonarSource's Java coding tasks, Opus 5.5 High's measurable task pass rate was less than one percentage point below Opus 5 Thinking's. Opus 5.5 generated less code and fewer issues overall, but had higher bug and concurrency issue densities per line of code.

Use cases

• Tasks this evaluation can inform: Functional pass rates for Java code generation, code volume, and bugs, security issues, and maintainability issues detectable by SonarQube.

• Tasks this evaluation should not be generalized to: Other languages, real production codebases, different prompts or toolchains, and issues not covered by SonarQube rules.

• Model versions covered: Claude Opus 5.5 High (described in the article as a pre-release version); comparator: Claude Opus 5 Thinking.

• Test environment or client: SonarSource's Sonar LLM evaluation framework. The Java benchmark covers HumanEval, MBPP, and ComplexCodeEval; generated code was analyzed by the SonarQube analyzer.

• Reasoning effort and parameters: Opus 5.5 used High. The article does not specify temperature, random seed, API/model build identifier, tool-call settings, output limit, or retry policy.

Evaluation method

• SonarSource says it used the Java benchmark and evaluation framework shared by all models on its LLM leaderboard.

• The Opus 5.5 run records 4,444 tasks across HumanEval, MBPP, and ComplexCodeEval. The functional pass rate counts only the 544 HumanEval and MBPP tasks with executable tests; ComplexCodeEval is used for code analysis and is not included in the pass rate.

• Generated code was evaluated by the SonarQube analyzer. Bug density is reported per million lines of code (mLOC), while code smell density and overall issue density are reported per thousand lines of code (kLOC).

• The methodology says the two models use the same category-rule mappings for vulnerabilities and code smells. Category rows are rounded to integers, so they may differ slightly from the aggregate densities.

• The article does not specify how the 4,444 tasks were distributed among the three suites, the prompts, task-level sample results, number of repeated runs, or statistical confidence intervals.

Key results

• The pass rates differ by 0.92 percentage points. Opus 5.5 generated 27.5% less code, used 40% fewer output tokens, and had 42% fewer total issues.

• Total bugs fell from 528 to 428, but bug density per mLOC rose from 576 to 644. Concurrency/threading bug density rose from 205 to 295 per mLOC, an increase of 44%, making it Opus 5.5's largest bug category.

• Total vulnerabilities fell from 230 to 152; vulnerability density fell from 251 to 229 per mLOC. Total code smells fell from 18,056 to 10,361.

• Density for all three highest-severity BLOCKER categories declined: reliability issues, 41→24; security issues, 19→9; and maintainability issues, 75→60 (all per mLOC).

• SonarSource reports that both runs used about 2.07M input tokens. Opus 5.5 recorded 3.23M reasoning tokens, while Opus 5 did not have this field recorded. The article says the older run likely did not collect this field, so it does not directly compare the two models' total token counts.

Raw data

The article reports the following aggregate metrics; density units are as shown in the table headers:

Opus 5.5 bug category densities (per mLOC): concurrency/threading 295, null/data value 92, performance/structure 71, resource/stream leaks 54, exception handling 41, type safety/casting 39, API contract violations 29, patterns/regular expressions 12, control flow errors 9, data structures 2, and uncategorized 2. The article notes that these values are rounded to integers.

Reported Opus 5.5 vulnerability category densities (per mLOC): cryptographic misconfiguration 104, insecure system resource handling 66, insufficient I/O error handling 23, injection attacks 17, hard-coded credentials 2, Web API security misconfiguration 5, path traversal/injection 5, XXE 3, missing certificate validation 0, and uncategorized 6. The article also explains that the rounded category values sum to 231, while the aggregate density is 229; each finding corresponds to about 1.5/mLOC.

Conclusions and limitations

This evaluation supports a limited conclusion: on the Java tasks selected by SonarSource and under its analyzer rules, Opus 5.5 High produced less code and fewer total detected issues than Opus 5 Thinking while keeping a broadly similar pass rate. The higher bug density and increase in concurrency-related issues show that less code does not mean every line is more reliable.

The results come from a benchmark built by SonarSource and analysis using SonarQube rules; they are not estimates of defect rates across languages or in real projects. The analyzer can detect only issues covered by its rules. The page does not provide task-level data, complete prompts, rule version, exact model build number, repeated samples, or confidence intervals, so readers cannot independently rerun the evaluation or assess the small pass-rate difference. Output token totals can be compared; reasoning-token reporting is asymmetric, so total tokens cannot be compared on that basis.

The page explicitly says the test used a pre-release build and that SonarSource would update its LLM Leaderboard results once Opus 5.5 reached general availability. These figures therefore should not be treated as a retest of the generally available release version.

Reproduction notes

The article discloses the model reasoning effort, language, benchmark suite names, total task count, pass-rate subset size, analyzer, and several aggregate results, but it does not provide task inputs, prompts, code artifacts, or the full run configuration. The table's figures and ratios can be checked against the article; reproducing the evaluation would still require SonarSource to provide the exact Java benchmark version and task list, prompts, model build and API parameters, SonarQube version, and raw task-level artifacts.