Benchmark: AA-Briefcase, Artificial Analysis’s private agentic knowledge-work benchmark.
Tasks: Complex, real-world-style input files, with deliverables including a spreadsheet, presentation, and UI mock-up.
Scoring: Elo synthesized from correctness, analytical quality, and presentation quality; the page does not publish the full tasks, prompts, or dataset.
Service: Kimi first-party API; pricing calculated at $3/$15 for Kimi K3 and $0.30/M for cached input.
The page does not publish the complete input, system prompt, tool list, or harness version for individual tasks.
It records turns, output tokens, cost, and time for each task, which can be used to assess task-level cost.
| Metric | Kimi K3 | Comparison/interpretation |
|---|---|---|
| AA-Briefcase Elo | 1543 | Second, behind only Fable 5 at 1574; above GPT-5.6 Sol at 1501 and Opus 4.8 at 1347 |
| Rubric pass rate | 51% | Behind only Fable 5 at 56% |
| Analytical quality Elo | 1754 | Close to Fable 5 at 1744 |
| Presentation quality Elo | 1471 | Below Sol at 1660 and Opus 4.8 at 1492 |
| Average cost/task | $10.57 | The page says this is roughly an order of magnitude higher than K2.6 |
| Average time/task | 56.4 minutes | About 2.5× Fable 5 and 3.8× Grok 4.5 high |
| Average output tokens | 120K | K2.6: 42K |
| Average turns | 83 | Fable 5: 67; Sol: 50 |
Kimi K3 is strong at analyzing complex source material and turning it into deliverables, but it trades more turns and longer outputs for that quality. If a workflow prioritizes “correct analysis” over visual presentation, K3 is worth trying. If every task must be delivered quickly, cheaply, and with polish, routing should account for time, presentation quality, and per-task cost.
AA-Briefcase is a private dataset, so readers cannot independently replay the original tasks; this document can verify the metrics and evaluation definition, but cannot claim full reproduction.
This is an agentic knowledge-work benchmark, not a representative test of short-form Q&A, code repair, or ordinary chat.
Cost and time depend on the first-party API, model version, tool calls, and task distribution; a single average cannot be used to infer every user’s bill.
A later snapshot in the Kimi team’s technical report lists AA-Briefcase Elo as 1548; this article’s snapshot is 1543. The date difference should be preserved rather than forcibly reconciled.
Build an in-house task set containing PDFs, spreadsheets, and presentation materials, and define separate correctness, analysis, and presentation rubrics.
Fix a first-party Kimi K3 API pricing snapshot, and record input/output tokens per turn, turn count, tool calls, total time, and human rework.
Give the same tasks to baseline models, blind-review the deliverables, and score the three categories separately.
Report means, percentiles, and cost per task; do not report only the final Elo.
The AA page publishes the benchmark’s task format and scoring dimensions, 1543 Elo, a 51% rubric pass rate, 1754 analytical Elo, 1471 presentation Elo, $10.57, 56.4 minutes, 120K output tokens, and 83 turns.
The page summary says: “second only to Fable 5 … but averaging nearly an hour per task”.
Kimi K3
