The author reports that LongCat 2.5 Preview was run three times on each of 89 TB 2.1 tasks, achieving a 48.7% Pass@1 rate and passing 64.0% of tasks at least once across three runs. This was an independent comparison with incomplete configuration details and raw run records.
Tasks this can help assess: Model comparisons on this specific TB 2.1 task set.
Tasks this should not be generalized to: Other coding tasks, different execution frameworks, or the overall capabilities of the released LongCat-2.5 version.
Applicable model version: The post title abbreviates the name as LongCat 2.5, but the data table explicitly identifies LongCat 2.5 Preview; these results should not be attributed directly to the released version.
Test environment or client: TB 2.1; the specific client, model provider, and execution environment are not stated.
Reasoning tier and parameters: Not stated.
The author says each model was run on the benchmark three times, for a total of 89 × 3 = 267 task runs. Pass@1 in the table uses all 267 runs as its denominator. Pass@3 uses the 89 tasks and means that a task passed at least once across three runs. The three models were run at different times; in a comment, the author says the MiMo results were reused from testing done the previous month.
Space Bunny had the highest Pass@1; the three models had similar Pass@3 results. LongCat 2.5 Preview had the fewest timeouts. The author also says LongCat had the fewest anomalies and timeouts, but the table columns are labeled Failures and Timeouts, and LongCat's Failures count is 86, higher than those of the other two models. The post does not explain how anomalies relate to Failures, so that statement cannot be interpreted as meaning LongCat had the fewest failures.
| Model | Pass@1 | Pass@3 | Failures | Timeouts |
|---|---|---|---|---|
| Space Bunny | 52.1% (139/267) | 66.3% (59/89) | 44 | 75 |
| LongCat 2.5 Preview | 48.7% (130/267) | 64.0% (57/89) | 86 | 48 |
| MiMo v2.5 | 44.2% (118/267) | 66.3% (59/89) | 85 | 58 |
LongCat's run window was 2026-10-04 22:20 to 2026-10-05 13:17 UTC. The author says their local device kept failing on the query-optimise task, so that task has no result in the task chart.
The data comes from one user, and the post does not provide complete run logs, specific configurations, or definitions of failures and timeouts. The three models were not run at the same time, and the MiMo data came from an earlier test. These results cannot be directly compared with benchmark scores obtained using different frameworks or conditions.
Use the same TB 2.1 task set and framework, run each selected model three times, record each result along with timeouts and failures, and aggregate using the denominators from the post. The post does not disclose enough configuration details or logs to reproduce these results strictly.
LongCat 2.5
