The author believes that model–harness fit (how well a model fits a framework) is real: the same model can perform very differently on different harnesses. After using OpenCode and Hermes to run DeepSeek V4 Flash in everyday work and seeing a clear cost difference, the author benchmarked DeepSeek V4 Flash (via OpenRouter) on 8 popular harnesses: 25 real-world automation tasks involving multiple applications (Slack, Sheets, Gmail, PostHog, and more).
| Harness | Pass rate | Median time | Tool calls | Cost per success |
|---|---|---|---|---|
| Pi Agent | 66.7% | 132.2s | 443 | $0.028 |
| Prime Agent | 62.5%* | 242.1s | 502 | $0.131 |
| OMP | 56.7% | 272.4s | 390 | $0.103 |
| Claude Code | 53.3% | 122.7s | 358 | $0.195 |
| Codex | 53.3% | 245.0s | 448 | $0.081 |
| DeepAgents | 53.3% | 187.1s | 353 | $0.045 |
| Hermes Agent | 50.0% | 175.5s | 386 | $0.056+ |
| OpenCode | 46.7% | 129.7s | 419 | $0.073 |
Pass rate and tool calls:
Pi Agent had the highest pass rate at 66.7%; OpenCode had the lowest at 46.7%.
More tool calls do not necessarily produce better results: DeepAgents (353 calls) and Codex (448 calls) each passed 16 tasks; OMP (390 calls) passed 17; OpenCode (419 calls) passed only 14.
Prime Agent passed 15 of 24 valid runs and made the most tool calls (502).
Cost and tokens:
Claude Code had the highest cost per successful run at $0.195; Pi had the lowest at $0.028.
Claude Code and OMP each used about 742K tokens per task, but Claude Code was nearly twice as expensive: its cache-hit rate was only 1.5% (Codex 70%, OMP 57%).
Prime Agent used 1.4M tokens per task, the most of any harness; Hermes used the fewest, at about 192K.
Time:
Claude Code had the shortest median time at 122.7s; OpenCode took 129.7s; Pi took 132.2s; OMP was the longest at 272.4s, but passed one more task than Claude Code.
Pi Agent is the best harness for DeepSeek V4 Flash: it had the highest accuracy and was the cheapest.
Claude Code is the biggest money burner.
Model–framework fit (cache utilization and tool-call efficiency) has a significant impact on real-world cost and results.
DeepSeek V4 Flash
