Using the official Kimi API as a reference, Fireworks found that production quality is determined by more than the model: K2.5's chat template, EOS/thinking boundary, null reasoning_content, sampling parameters, and load errors can all change tool-call results, while Fireworks' representative scores are close to the official figures with minor differences.
Comparison: Kimi model card, Kimi official API measured by Fireworks, and Fireworks API.
Quality layer: prompt formatting, stream/non-stream tool calling, grammar-constrained generation, numerical correctness, timeout/disconnect/error codes, and SDK/proxy compatibility.
Evaluation layer: deterministic unit tests, one-turn, agentic multi-turn, multimodal, and periodic regression; KVV, SWE-Agent, Terminus-2, and Eval Protocol.
Repeats: Most results avg@3; Tau-Bench avg@4; SWE-Bench avg@2.
The Fireworks report states that SWE-Bench uses SWE-Agent, Terminal-Bench 2 uses Terminus 2, and AIME/MMMU/OCR/K2VV use Kimi KVV. For SWE-Bench, Fireworks raised max output to 32k, while the official harness does not specify max_tokens; Terminal-Bench is forced to non-thinking mode.
| Evaluation | Kimi Model Card | Official API (measured by Fireworks) | Fireworks API |
|---|---|---|---|
| SWE-Bench | 76.8 | 76.8 | 76.8 |
| Terminal-Bench 2 (non-thinking) | 50.8 | Not publicly disclosed | 50.6 |
| Tau2 Airline | Not publicly disclosed | 65 | 68 |
| AIME 2025 | 96.1 | 95.7 | 95.0 |
| K2VV ToolCall F1 (non-thinking) | 84.0 | Not publicly disclosed | 83.9 |
| MMMU Pro Vision | 77.4 | Not publicly disclosed | 77.9 |
| OCRBench | 91.0 | Not publicly disclosed | 91.7 |
Fireworks' measurements support the view that “correct deployment can approach official results,” but they also show that production systems must handle thinking-phase EOS, null reasoning_content, sampling parameters, empty 200 responses, and multi-turn tool calls. Product selection should evaluate the complete serving stack rather than compare model cards alone.
Fireworks has a provider perspective; code/fix details and some official KVV results depend in part on its own service.
Most scores use only a small number of repeats (2–4) and cannot represent variance across all tasks.
Promotional comparisons such as “1/10 the cost” and “2–3× the speed” depend on provider conditions and are not universal guarantees of price or throughput.
Terminal non-thinking, the SWE custom harness, and official API conditions are not identical; cross-column comparisons require caution.
For the same prompt, model snapshot, tool schema, sampling settings, and max tokens, call the official API, Fireworks, and a self-hosted endpoint separately.
Run single-turn/multi-turn, stream/non-stream, Thinking/Instant, vision, and grammar-constrained tool tests.
Verify the chat template, reasoning_content null filtering, EOS guard, HTTP 429/empty 200 responses, timeouts, and retries.
Report score, call success rate, error type, latency, and cost for each harness; avoid summarizing deployment quality with a single SWE score.
Kimi K2.5
