Strong and cheap is incomplete: community disagreement on cost and capability

Original article

Skip to main content Kimi K3 Is Impressive, but "Better and Much Cheaper" Is Too Simplistic : r/LLMDevs Advertise on Reddit Open chat Create Create post Open inbox Expand user menu Repost Go to “LLMDevs” r/LLMDevs • 22 days ago s243a Kimi K3 Is Impressive, but "Better and Much Cheaper" Is Too Simplistic Kimi K3 Is Impressive, but “Better and Much Cheaper” Is Too Simplistic Discussion

Kimi K3 is getting a lot of hype. Some claims say it beats Fable 5, GPT-5.6 Sol, even Opus 5. I don't buy the strong version. My read: Kimi K3 sits between the previous frontier tier (Opus 4.8 / GPT-5.5) and the current one (Fable 5 / GPT-5.6 Sol), genuinely good, but not quite there. On Artificial Analysis's Intelligence Index, Kimi scores 57, behind both Fable 5 and GPT-5.6 Sol, roughly level with Opus 4.8 and GPT-5.5. x Kimi K3 is getting a lot of hype. Some claims say it beats Fable 5, GPT-5.6 Sol, and even Opus 5. I don't buy the strong version. My read: Kimi K3 sits between the previous frontier tier (Opus 4.8 / GPT-5.5) and the current one (Fable 5 / GPT-5.6 Sol), genuinely good but not quite there. On Artificial Analysis's Intelligence Index, Kimi scores 57, behind both Fable 5 and GPT-5.6 Sol, roughly level with Opus 4.8 and GPT-5.5.

The benchmark headline problem The benchmark headline problem

"Kimi beats Fable at X" often hides which X: frontend generation, a specific harness, an effort setting, or pass@k with multiple attempts allowed. DeepSWE shows this clearly, and the cost evidence here is genuinely mixed. "Kimi beats Fable at X" often hides which X: frontend generation, a specific harness, an effort setting, or pass@k with multiple attempts allowed. DeepSWE shows this clearly, and the cost evidence here is genuinely mixed.

In one Kimi K3 Max vs GPT-5.6 Sol Max comparison, Sol wins pass@1 (72.7% vs 68.5%), but Kimi is cheaper per rollout ($4.65 vs $8.37) and pulls ahead at higher pass@k. A separate small programming micro-benchmark found Sol cheaper per correct answer than Kimi — but that wasn't DeepSWE, so it shouldn't be generalized. These aren't necessarily contradictory; they measure different things: one high-confidence attempt vs several cheap ones, cost-per-rollout vs cost-per-correct-solve. Anyone citing a single DeepSWE cost number without specifying which is skipping the part that matters. linkedin In one Kimi K3 Max vs GPT-5.6 Sol Max comparison, Sol wins pass@1 (72.7% vs 68.5%), but Kimi is cheaper per rollout ($4.65 vs $8.37) and pulls ahead at higher pass@k. A separate small programming micro-benchmark found Sol cheaper per correct answer than Kimi — but that wasn't DeepSWE, so it shouldn't be generalized. These aren't necessarily contradictory; they measure different things: one high-confidence attempt vs several cheap ones, cost-per-rollout vs cost-per-correct-solve. Anyone citing a single DeepSWE cost number without specifying which is skipping the part that matters. linkedin

Why I still rank it below Why I still rank it below

Interesting programming pulls from math, algorithms, systems tradeoffs, and domain knowledge outside the codebase. That's why broader reasoning benchmarks matter even for coding. They're a proxy for whether a model c an transfer concepts when a task isn't "edit this function" but "figure out the right approach first." Interesting programming pulls from math, algorithms, systems tradeoffs, and domain knowledge outside the codebase. That's why broader reasoning benchmarks matter even for coding. They're a proxy for whether a model can transfer concepts when a task isn't "edit this function" but "figure out the right approach first."

The gap here is concrete. Fable 5 scored 88% on FrontierMath Tier 4, about 13 points above GPT-5.5's ~75%. Artificial Analysis also has Fable 5 leading its AA-Omniscience knowledge benchmark. GPT-5.6 Sol trails Fable by roughly a point on the aggregate Intelligence Index while costing about a third as much, and it topped GeneBench-Pro, a hard genomics/quantitative-biology benchmark, at 31.5% — a decent proxy for general scientific reasoning, if not coding directly. aiweekly The gap here is concrete. Fable 5 scored 88% on FrontierMath Tier 4, about 13 points above GPT-5.5's ~75%. Artificial Analysis also has Fable 5 leading its AA-Omniscience knowledge benchmark. GPT-5.6 Sol trails Fable by roughly a point on the aggregate Intelligence Index while costing about a third as much, and it topped GeneBench-Pro, a hard genomics/quantitative-biology benchmark, at 31.5%—a decent proxy for general scientific reasoning, if not coding directly. aiweekly

Kimi K3 doesn't show up as a contender on any of these. Its strengths sit in a different lane: frontend generation, some agentic coding, not the deep cross-domain reasoning the newest tier is winning on. That's the real basis for ranking it below Fable 5 and GPT-5.6 Sol: not just index position, but a measured gap in the cross-disciplinary reasoning that separates "good coding agent" from "frontier model." Kimi K3 doesn't show up as a contender on any of these. Its strengths sit in a different lane: frontend generation, some agentic coding, not the deep cross-domain reasoning the newest tier is winning on. That's the real basis for ranking it below Fable 5 and GPT-5.6 Sol: not just index position, but a measured gap in the cross-disciplinary reasoning that separates "good coding agent" from "frontier model."

API price ≠ task price API price ≠ task price

Kimi's tokens are cheap ($3/$15 per million vs Sol's $5/$30). But cheaper tokens don't guarantee cheaper tasks — longer runs, more turns, more retries eat the margin. Artificial Analysis found Kimi and Sol nearly tied on cost per task ($0.94 vs $1.04), despite the sticker-price gap. My guess: Kimi's edge holds on short, easy, cache-friendly work, and shrinks as tasks get harder. myclaw Kimi's tokens are cheap ($3/$15 per million vs Sol's $5/$30). But cheaper tokens don't guarantee cheaper tasks—longer runs, more turns, more retries eat the margin. Artificial Analysis found Kimi and Sol nearly tied on cost per task ($0.94 vs $1.04), despite the sticker-price gap. My guess: Kimi's edge holds on short, easy, cache-friendly work, and shrinks as tasks get harder. myclaw

Subscriptions are murkier still Subscriptions are murkier still

I burned 6.87% of my monthly Moderato quota in a few hours doing GitHub-connected code review. That's not a controlled benchmark, just one real data point. Kimi's docs confirm Agent, Deep Research, Kimi Code, and connectors all draw from one shared cr edit pool metered by token use. A $19/month price tells you little about how far that actually goes in real agentic work. kimi I burned 6.87% of my monthly Moderato quota in a few hours doing GitHub-connected code review. That's not a controlled benchmark, just one real data point. Kimi's docs confirm Agent, Deep Research, Kimi Code, and connectors all draw from one shared credit pool metered by token use. A $19/month price tells you little about how far that actually goes in real agentic work. kimi

One aside: engineer vs. scientist One aside: engineer vs. scientist

Subjectively, Claude tends to commit to a complete implementation in one pass; GPT/Codex explores well but often needs more "continue" prompts to finish. That changes effective cost because finishing in one shot beats needing three follow-ups, even at a higher sticker price.

Bottom line

Kimi K3 is a legitimately strong near-frontier model, likely the better economic choice for easy-to-medium tasks. But "clearly better than Fable/Sol" and "obviously much cheaper" both overstate the evidence. DeepSWE cost comparisons point in different directions depending on setup — that's the actual state of the data, not a gap in this analysis. What would change my mind: a larger, harness-controlled study measuring cost-per-correct-completion across a real mix of easy and hard tasks.

Share RedditforBusiness • Promoted "I mean, I'd LOVE to try running ads on Reddit, but it's not like you can target by subreddit!!!!" Sign up ads.reddit.com Sort by: Comments Recoil42 • 22 days ago

Thanks Claude.

s243a • 22 days ago

Here is the thread that used 6.7% of my monthly usage (only pat of the conversation displayed):

https://www.kimi.com/share/19f9c2e8-52a2-84d5-8000-000090bb69e7

The link should give the full conversation. Here is part of it:

Prompt

GitHubFirst review the UnifyWeaver PR, "mu_cosine: filing path decoder Stage A + process-expression contract fixtures #4… This is a necessary excerpt; read the original source for full context.

I'll load the GitHub tools and pull up that PR. Found the repo (s243a/UnifyWeaver). Now let me pull up PR #4000. Got the PR description. Now let me look at the actual changes — the file list, CI status, and the diff itself. I've gone through the PR in full — the description, all 9 changed files (5,760 additions), and the CI runs (all green). Here's the explanation first, then my review.

What the PR description is actually saying

The setting

prototypes/mu_cosine is a research prototype inside UnifyWeaver that studies ML-assisted filing: given a bookmark and a frozen, previously-ranked menu of candidate folders (ranked by a frozen "e5" embedding model), decide where in a folder hierarchy it should be ecide where in a folder hierarchy it should be filed. The project is run with an unusually strict provenance/scientific-validity discipline: every artifact is hash-bound, every decision is advisory and receipted, and anything that smells like data leakage or unjustified automation is designed to fail closed (raise an error rather than degrade gracefully).

Two design documents govern this PR, and the PR's whole point is that it implements exactly what those documents authorize as the first engineering step — and nothing more:

DESIGN_filing_path_decoder_handoff.md §11 — the filing path decoder

DESIGN_expression_encoder_future.md §11 — the process-expression encoder (a planned learned encoder for "process expressions," the little DSL strings like e5(routing(e5,haiku,t=[0.02],menus=[10])) that describe composed scoring/routing pipelines)

Part 1 — "Stage A of the filing path decoder"

The decoder is planned in stages. Stage A is the most conservative possible slice: it never creates folders, never mutates the graph, never calls an external API, and can only emit two outcomes:

SELECT_EXISTING — "file it in this existing folder," where the folder is simply the first eligible candidate in the parent task's frozen ranking order (no re-scoring, no re-ranking), and

ABSTAIN — with a typed reason code (ambiguous_existing, no_eligible_candidate, resource_censored, …).

...

No_Inspection4415 • 17 days ago

Is Opus 5 a better model in your opinion?

s243a • 16 days ago

I think the models that are better than kimi k3 are Fable 5, Opus 5 and GPT 5.6 dol. That said, I've only been using kimi to explain to me the stuff that claude or gpt are doing, kind of like a reviewer. I do this using the github plugin in the chat. I find kimi k3 as a reviewer is better than perplexity but I don't have access to the top models on my perplexity plan. Perplexity computer is impressive but costly.

I saw a benchmark agregate saying that Grok 4.5 is better than kimi k3. I'm suspicious of that result. For one thing I don't like Grok's writing style, and I only use Grok as a worker under the direction of gpt or claude. I don't think grok's even as good as opus 4.8 but that's probably a vibes based opinion since I know grok 4.5 has it's fans.

While I use gpt claude and kimi a fair amount, I feel my opinion on kimi is somewhat vibes based because I haven't used kimi yet as a coding agent.

habeeb-s • 7 days ago

What models do you you use for what kind of tasks and why?

Created February 17, 2023 Public 60K 1,497 User flair 1llegi

3 years, February 17 Public 60K 1,497 User flair 1llegi Community bookmarks Wiki R/LLMDEVS rules 1 Be respectful: No personal attacks or harassment. Be respectful: No personal attacks or harassment. 2 Ask ethically: Disclose purpose of questions and surveys. Ask ethically: Disclose purpose of questions and surveys. 3 Stay accurate: Avoid false or misleading information. Stay accurate: Avoid false or misleading information. 4 Stay on topic: Related to LLM and NLP. Stay on topic: Related to LLM and NLP. 5 No commercial self-promotion: Share openly, not for profit. No commercial self-promotion: Share openly, not for profit. 6 No low-effort posts or spam: High-quality content only. No low-effort posts or spam: High-quality content only. 7 Provide sources and references: Cite sources for accuracy. Provide sources and references: Cite sources for accuracy. 8 No NSFW content: Keep it professional. No NSFW content: Keep it professional. 9 Follow Reddit's rules: Follow Reddit's content policy and rules. Follow Reddit's rules: Follow Reddit's content policy and rules. 10 Be genuine: No disguised advertising or marketing Be genuine: No disguised advertising or marketing Moderators Message the moderators u/m2845 u/Scipio_Afri u/h8mx u/robogame_dev robogame u/purge-user View all moderators Installed apps Remove Macro Reddit Rules Privacy Policy User Agreement Your Privacy Choices Accessibility Reddit, Inc. © 2026. All rights reserved. Collapse navigation Create a community Games on Reddit Customize feed Create custom feed Recent r/opencodeCLI r/chrome r/Notion r/todoist Communities Manage communities Resources About Reddit Advertise Developer platform Reddit Pro Beta Help Blog Careers Press Reddit Best Reddit Rules Privacy Policy User Agreement Your Privacy Choices Accessibility Reddit, Inc. © 2026. All rights reserved.

undefined