Harvey recorded Claude Sonnet 4.6 at a 4.2% full-pass rate on the legal agent benchmark LAB, below Opus 4.6 at 6.6%; on the same leaderboard, post-trained NVIDIA Nemotron 3 Ultra reached 5.8%, and claimed operating costs are 1/8 to 1/50 of Sonnet/Opus.
Benchmark: Harvey Legal Agent Bench (LAB).
Metric: full-pass / full-pass rate.
Comparison: Nemotron 3 Ultra baseline 0% → post-training 5.8%; Sonnet 4.6 4.2%; Opus 4.6 6.6%.
Additional observation: held-out tasks were ~70% pass before training (because enough scoring dimensions were missed), ~95% after training. The 70%/95% refers to Nemotron before/after post-training, not Sonnet.
The X post did not disclose LAB questions, scoring dimensions, Sonnet's prompt, toolset, effort, or whether legal retrieval plugins were enabled. The Trajectory post stated the entire post-training was completed in under 24 hours after Nemotron 3 Ultra's release.
| Model | LAB full-pass rate |
|---|---|
| Nemotron 3 Ultra (no post-training) | 0% |
| Claude Sonnet 4.6 | 4.2% |
| Nemotron 3 Ultra (legal post-training) | 5.8% |
| Claude Opus 4.6 | 6.6% |
Harvey also wrote: the post-trained open-weight model reached quality close to leading closed-source models, with operating costs at 1/8 to 1/50 of Sonnet 4.6 and Opus 4.6 per-token prices.
Legal agent "full pass" is very strict: Sonnet 4.6's 4.2% does not mean it cannot do legal assistance, but means that under Harvey's all-dimensions-pass standard, it is clearly weaker than Opus 4.6 and also weaker than a specially post-trained Nemotron. Sonnet 4.6 is suitable for legal drafting/retrieval assistance, not as an automatic pass agent in LAB terms. High-risk legal deliverables should still go through Opus or domain post-trained models, with lawyer review retained.
Full-pass rate is not partial correctness rate; 4.2% is not the same metric as everyday "can write a usable memo."
Per-question results and Sonnet configuration not disclosed.
Cost 1/8–1/50 is the author's statement on Nemotron vs Claude unit prices, not LAB scores.
Cannot conflate LAB with IDP document extraction or SWE coding.
If Harvey/Trajectory later publish a LAB subset, fix the same scoring dimensions and run claude-sonnet-4-6 vs claude-opus-4-6.
Report full pass, partial pass, citation errors, and missed scoring dimensions separately.
Record retrieval tools, jurisdictions, and whether internet access was allowed.
Legal conclusions must be reviewed by qualified personnel; model scores cannot substitute.
Claude Sonnet 4.6
