GPT-5.6 Terra: SonarSource's Retest of Code Quality and Security on 4,444 Java Tasks

One-sentence takeaway

On the same set of 4,444 Java tasks and with the medium reasoning configuration, Terra generated shorter code and had fewer missing completions, but its cognitive complexity and bug/vulnerability and code smell densities were higher, making static analysis and testing essential.

Test environment

• Models: GPT-5.6 Sol, GPT-5.6 Terra, and GPT-5.5.

• Language: Java.

• Number of tasks: 4,444, sourced from HumanEval, MBPP, and ComplexCodeEval.

• Reasoning setting: medium for all three models.

• Analyzer: SonarQube algorithmic code analysis.

• Metric units: complexity/code smell per kLOC; bugs/vulnerabilities and category breakdowns per mLOC.

Inputs/configuration

SonarSource says the three models used the same tasks, the same quality analysis, and the same reasoning tier, and that GPT-5.5 was rerun in the same framework. The article does not disclose the prompt for each question, snapshots, sampling parameters, complete generated outputs, or a task-level failure list.

Results

Additional data: Terra produced approximately 8.37M output tokens, including approximately 3.66M reasoning tokens; input was approximately 1.32M tokens. Terra's blocker-level smell rate was 62/mLOC, lower than Sol's 72 and GPT-5.5's 78, but collection/generics issues rose to 11,843/mLOC. Concurrency/threading bugs were Terra's largest bug category, at approximately 350/mLOC.

Conclusions

• Terra's 79.96% pass rate was slightly higher than GPT-5.5's but lower than Sol's 81.99%; production reliability cannot be judged by model price alone.

• Terra completed the same task with 617,132 lines of code, substantially less than GPT-5.5 and Sol; this helps reduce review volume, but the problem density per kLOC was higher.

• Cognitive complexity of 161.53/kLOC, bug density of 763/mLOC, vulnerability density of 203/mLOC, and smell density of 23.31/kLOC show that “shorter” does not mean “easier to maintain.”

• Concurrency/threading, cryptographic configuration, and resource handling are risk areas that should receive priority in automated checks; manual read-through should not be the only gate.

Limitations

• All tasks were synthetic/standardized Java tasks and cannot represent real multilingual repositories.

• SonarQube static analysis reflects detectable code issues; it is not equivalent to the true online defect rate and cannot measure the business correctness of a complete system.

• The same medium setting enables a controlled comparison, but the article does not tell us Terra's quality-cost curve at low/high/max settings.

• SonarSource published the article and controlled the analyzer and metric definitions; retain the original outputs and cross-validate with a second testing framework.

Reproduction steps

1. Fix the Java version, HumanEval/MBPP/ComplexCodeEval task versions, model aliases, and the medium reasoning tier.

2. Save the prompt, response, compilation/test logs, generated code line count, and tokens for each task; do not save only aggregate scores.

3. Analyze all three output groups with the same SonarQube version and rule set, calculating densities in the article's kLOC/mLOC units.

4. Report pass rate, missing completions, and bugs/vulnerabilities/smells separately, broken down by severity.

5. Manually sample concurrency, cryptography, and resource-handling issues to confirm whether static-rule hits constitute real defects.

Source excerpt or observation (for a compliant short quotation only)

The authors' summary of Terra is “Terra is the concise one”, but the same data shows that its cognitive complexity and problem density were the highest.