Knowledge-work quality is competitive, but cost and time per task are high

Test environment

• Benchmark: AA-Briefcase, Artificial Analysis’s private agentic knowledge-work benchmark.

• Tasks: Complex, real-world-style input files, with deliverables including a spreadsheet, presentation, and UI mock-up.

• Scoring: Elo synthesized from correctness, analytical quality, and presentation quality; the page does not publish the full tasks, prompts, or dataset.

• Service: Kimi first-party API; pricing calculated at $3/$15 for Kimi K3 and $0.30/M for cached input.

Input/configuration

• The page does not publish the complete input, system prompt, tool list, or harness version for individual tasks.

• It records turns, output tokens, cost, and time for each task, which can be used to assess task-level cost.

Results

Conclusion

Kimi K3 is strong at analyzing complex source material and turning it into deliverables, but it trades more turns and longer outputs for that quality. If a workflow prioritizes “correct analysis” over visual presentation, K3 is worth trying. If every task must be delivered quickly, cheaply, and with polish, routing should account for time, presentation quality, and per-task cost.

Limitations

• AA-Briefcase is a private dataset, so readers cannot independently replay the original tasks; this document can verify the metrics and evaluation definition, but cannot claim full reproduction.

• This is an agentic knowledge-work benchmark, not a representative test of short-form Q&A, code repair, or ordinary chat.

• Cost and time depend on the first-party API, model version, tool calls, and task distribution; a single average cannot be used to infer every user’s bill.

• A later snapshot in the Kimi team’s technical report lists AA-Briefcase Elo as 1548; this article’s snapshot is 1543. The date difference should be preserved rather than forcibly reconciled.

Reproduction steps

1. Build an in-house task set containing PDFs, spreadsheets, and presentation materials, and define separate correctness, analysis, and presentation rubrics.

2. Fix a first-party Kimi K3 API pricing snapshot, and record input/output tokens per turn, turn count, tool calls, total time, and human rework.

3. Give the same tasks to baseline models, blind-review the deliverables, and score the three categories separately.

4. Report means, percentiles, and cost per task; do not report only the final Elo.

Original evidence and data

The AA page publishes the benchmark’s task format and scoring dimensions, 1543 Elo, a 51% rubric pass rate, 1754 analytical Elo, 1471 presentation Elo, $10.57, 56.4 minutes, 120K output tokens, and 83 turns.

Source excerpt or observation (short compliance quote only)

The page summary says: “second only to Fable 5 … but averaging nearly an hour per task”.