Kimi K2.5 · Media / benchmark · Independent measurement
Fireworks reruns Kimi through the official API and shows that chat templates, EOS, reasoning_content, sampling, and load errors can change tool-call quality.
Unverified: the original source could not be rechecked. Historical figures below are not current verified results.
Using the official Kimi API as a reference, Fireworks found that production quality is determined by more than the model: K2.5's chat template, EOS/thinking boundary, null reasoning_content, sampling parameters, and load errors can all change tool-call results, while Fireworks' representative scores are close to the official figures with minor differences.
Comparison: Kimi model card, Kimi official API measured by Fireworks, and Fireworks API.
Quality layer: prompt formatting, stream/non-stream tool calling, grammar-constrained generation, numerical correctness, timeout/disconnect/error codes, and SDK/proxy compatibility.
Evaluation layer: deterministic unit tests, one-turn, agentic multi-turn, multimodal, and periodic regression; KVV, SWE-Agent, Terminus-2, and Eval Protocol.
Repeats: Most results avg@3; Tau-Bench avg@4; SWE-Bench avg@2.
The Fireworks report states that SWE-Bench uses SWE-Agent, Terminal-Bench 2 uses Terminus 2, and AIME/MMMU/OCR/K2VV use Kimi KVV. For SWE-Bench, Fireworks raised max output to 32k, while the official harness does not specify max_tokens; Terminal-Bench is forced to non-thinking mode.
| Evaluation | Kimi Model Card | Official API (measured by Fireworks) | Fireworks API |
|---|---|---|---|
| SWE-Bench | 76.8 | 76.8 | 76.8 |
| Terminal-Bench 2 (non-thinking) | 50.8 | Not publicly disclosed | 50.6 |
| Tau2 Airline | Not publicly disclosed | 65 | 68 |
| AIME 2025 | 96.1 | 95.7 | 95.0 |
| K2VV ToolCall F1 (non-thinking) | 84.0 | Not publicly disclosed | 83.9 |
| MMMU Pro Vision | 77.4 | Not publicly disclosed | 77.9 |
| OCRBench | 91.0 | Not publicly disclosed | 91.7 |
Fireworks' measurements support the view that “correct deployment can approach official results,” but they also show that production systems must handle thinking-phase EOS, null reasoning_content, sampling parameters, empty 200 responses, and multi-turn tool calls. Product selection should evaluate the complete serving stack rather than compare model cards alone.
Fireworks has a provider perspective; code/fix details and some official KVV results depend in part on its own service.
Most scores use only a small number of repeats (2–4) and cannot represent variance across all tasks.
Promotional comparisons such as “1/10 the cost” and “2–3× the speed” depend on provider conditions and are not universal guarantees of price or throughput.
Terminal non-thinking, the SWE custom harness, and official API conditions are not identical; cross-column comparisons require caution.
For the same prompt, model snapshot, tool schema, sampling settings, and max tokens, call the official API, Fireworks, and a self-hosted endpoint separately.
Run single-turn/multi-turn, stream/non-stream, Thinking/Instant, vision, and grammar-constrained tool tests.
Verify the chat template, reasoning_content null filtering, EOS guard, HTTP 429/empty 200 responses, timeouts, and retries.
Report score, call success rate, error type, latency, and cost for each harness; avoid summarizing deployment quality with a single SWE score.
The figures, task set, reasoning tier, and client conditions apply only to the listed source and collection snapshot. Different versions, harnesses, or providers must not be compared directly; undisclosed parameters remain unknown.
For a reproduction, fix the model version, provider or client, reasoning tier, tools, task-set version, sample count, and collection date, and record failures, retries, and human corrections. Full steps are in the source notes below.
Fireworks AI · Fireworks AI · Original publication date Unknown · Site edit date 2026-09-20
Open original sourceKimi K2.5
Download the Tabbit client to check model access
