DeepSeek V4 Flash · Community source · Independent measurement
A Reddit author compares eight harnesses on OpenRouter across 25 automation tasks: Pi Agent passes 66.7% versus OpenCode 46.7%, with about $0.028 versus $0.073 per successful task.
The author believes that model–harness fit (how well a model fits a framework) is real: the same model can perform very differently on different harnesses. After using OpenCode and Hermes to run DeepSeek V4 Flash in everyday work and seeing a clear cost difference, the author benchmarked DeepSeek V4 Flash (via OpenRouter) on 8 popular harnesses: 25 real-world automation tasks involving multiple applications (Slack, Sheets, Gmail, PostHog, and more).
| Harness | Pass rate | Median time | Tool calls | Cost per success |
|---|---|---|---|---|
| Pi Agent | 66.7% | 132.2s | 443 | $0.028 |
| Prime Agent | 62.5%* | 242.1s | 502 | $0.131 |
| OMP | 56.7% | 272.4s | 390 | $0.103 |
| Claude Code | 53.3% | 122.7s | 358 | $0.195 |
| Codex | 53.3% | 245.0s | 448 | $0.081 |
| DeepAgents | 53.3% | 187.1s | 353 | $0.045 |
| Hermes Agent | 50.0% | 175.5s | 386 | $0.056+ |
| OpenCode | 46.7% | 129.7s | 419 | $0.073 |
Pass rate and tool calls:
Pi Agent had the highest pass rate at 66.7%; OpenCode had the lowest at 46.7%.
More tool calls do not necessarily produce better results: DeepAgents (353 calls) and Codex (448 calls) each passed 16 tasks; OMP (390 calls) passed 17; OpenCode (419 calls) passed only 14.
Prime Agent passed 15 of 24 valid runs and made the most tool calls (502).
Cost and tokens:
Claude Code had the highest cost per successful run at $0.195; Pi had the lowest at $0.028.
Claude Code and OMP each used about 742K tokens per task, but Claude Code was nearly twice as expensive: its cache-hit rate was only 1.5% (Codex 70%, OMP 57%).
Prime Agent used 1.4M tokens per task, the most of any harness; Hermes used the fewest, at about 192K.
Time:
Claude Code had the shortest median time at 122.7s; OpenCode took 129.7s; Pi took 132.2s; OMP was the longest at 272.4s, but passed one more task than Claude Code.
Pi Agent is the best harness for DeepSeek V4 Flash: it had the highest accuracy and was the cheapest.
Claude Code is the biggest money burner.
Model–framework fit (cache utilization and tool-call efficiency) has a significant impact on real-world cost and results.
The figures, task set, reasoning tier, and client conditions apply only to the listed source and collection snapshot. Different versions, harnesses, or providers must not be compared directly; undisclosed parameters remain unknown.
For a reproduction, fix the model version, provider or client, reasoning tier, tools, task-set version, sample count, and collection date, and record failures, retries, and human corrections. Full steps are in the source notes below.
Reddit r/DeepSeek · u/LimpComedian1317 · Original publication date Unknown · Site edit date 2026-09-20
Open original sourceDeepSeek V4 Flash
Download the Tabbit client to check model access
