In two tests of its code review pipeline, CodeRabbit found some issues that its production baseline missed, while also missing some issues the baseline caught. Standard slightly outperformed Max on 80 OSS patterns; on 13 harder Signal cases, Max produced more actionable hits, but also more comments and higher token usage.
Tasks this can help assess: Compare known-defect detection, actionable precision, comment volume, and the trade-offs between Standard and Max configurations in CodeRabbit's PR review workflow.
Tasks this should not be generalized to: The results do not establish general code-generation ability, performance in other review tools, or whether production teams would accept these comments. Signal has only 13 cases, which is too few to represent a broad distribution of software defects.
Applicable model version: Claude Opus 5.5; Standard and Max are configurations in CodeRabbit's pipeline, not single API effort values or Anthropic defaults.
Test environment or client: CodeRabbit's own code review pipeline, compared with its production model combination (production baseline).
Reasoning level and parameters: Standard combines a lower reasoning effort with the pipeline; Max combines a higher effort. The source does not disclose the complete parameter mapping for each stage.
CodeRabbit ran the same review pipeline configurations on 80 shared known-defect patterns from the OSS August benchmark, and tested 13 harder cases on the separate Signal benchmark. Each set compared the production baseline, Opus 5.5 Standard, and Opus 5.5 Max. The results were validated, deduplicated, and filtered.
Reported metrics include known-issue hits, actionable precision, and comment volume. In the article, “precision” means the share of comments about target issues that were confirmed by the benchmark adjudicator; it is not the developer acceptance rate. Actionable results count only actionable findings on changed lines; full-stream also includes findings outside changed lines. Major and Minor are labels the model assigned to comments, not counts of distinct defects.
OSS August: 80 patterns
| Configuration | Actionable recall | Actionable precision | Reported comments |
|---|---|---|---|
| Production baseline | 49/80 · 61.3% | 39.3% | 116 |
| Opus 5.5 Standard | 51/80 · 63.8% | 38.6% | 127 |
| Opus 5.5 Max | 50/80 · 62.5% | 35.7% | 140 |
Relative to the production baseline in the same test, Standard increased actionable recall by 2.50 percentage points and reduced precision by 0.7 percentage points; Max increased recall by 1.25 percentage points and reduced precision by 3.6 percentage points. Standard found 11 patterns missed by the baseline, while missing 9 that the baseline caught; Max found 14 new patterns, while missing 13 caught by the baseline. All three missed another 20 patterns. The source says both Opus 5.5 configurations found the concurrent update issue in Cal.com's retry counter, which the production baseline missed.
Signal: 13 harder patterns
| Configuration | Actionable hits | Actionable precision | Reported comments | Major comments | Minor comments |
|---|---|---|---|---|---|
| Production baseline | 5/13 · 38.5% | 29.4% | 17 | 11 | 6 |
| Opus 5.5 Standard | 8/13 · 61.5% | 66.7% | 21 | 14 | 7 |
| Opus 5.5 Max | 10/13 · 76.9% | 52.0% | 25 | 13 | 12 |
Including findings outside changed lines, the baseline covered 7/13 cases, while Standard and Max each covered 10/13. Standard and Max found different sets of issues in their full reviews. The source does not provide a per-case hit list for the two configurations, so the article does not allow readers to calculate the actual combined effect of using both reviews.
Token usage
| Configuration | OSS August: relative to production baseline | Signal: relative to production baseline |
|---|---|---|
| Opus 5.5 Standard | +49.2% | +40.6% |
| Opus 5.5 Max | +57.6% | +60.1% |
These are changes in token usage from the run summaries; the source does not break out input, output, or cached tokens, nor does it provide corresponding dollar costs or latency. The article also lists model prices: $4 per million input tokens, $20 per million output tokens, and $0.20 per million cached-read tokens. It says these are lower than Opus 5's prices of $5, $25, and $0.50, respectively. A change in list prices does not directly establish that the total cost of a review is lower.
In these two test sets, Opus 5.5 produced a different combination of defect hits from the baseline. Coverage gains on OSS August were modest and came with lower actionable precision and more comments.
On OSS August, Standard achieved a slightly higher hit count and precision with fewer comments, making it a reasonable initial candidate for CodeRabbit's self-evaluation. On Signal, Max had more hits on changed lines, but lower precision than Standard and more comments; when findings outside changed lines are included, Standard and Max both hit 10/13.
The article presents historical Opus 5 results alongside Opus 5.5 as context, but the two rounds used different test sets and configurations. They should not be read as a direct comparison of the improvement between model versions.
This is CodeRabbit's evaluation of its own product pipeline. The public article does not provide the complete lists of the 80 and 13 test samples, adjudicator details, raw records for each comment, or run scripts; the test cannot be independently rerun from the article. The results also do not measure the cost, comment volume, or practical benefit of running both reviewers together.
These conclusions apply to the benchmarks and pipeline described in the article. Teams evaluating the model should check newly found and missed issues in their own codebases, while also tracking review burden, token categories, cost, and latency.
To verify the source, review “What we tested,” “Start your evaluation with Standard,” “Test what higher effort changes,” and the pricing section. Reproducing a comparable evaluation requires the same versions of the 80 OSS patterns and 13 Signal cases, the production baseline and both CodeRabbit configurations, the adjudicator and filtering rules, and records of findings inside and outside changed lines and a complete token breakdown. The article does not fully disclose these materials.
Claude Opus 5.5
