SlopCodeBench: Sol ties Fable on strict long-horizon coding passes

Test environment

• Benchmark: SlopCodeBench. The tasks do not reveal all requirements at once; they add requirements progressively across multiple checkpoints to test the risk of codebase degradation over time.

• Scale: 6 challenges and 30 checkpoints, with a fresh context for each checkpoint; each of the four models was run once.

• Harness: Fable—Claude Code 2.1.219; Sol—Codex CLI 0.145.0; Kimi K3 (Baseten/Modal)—OpenCode 1.18.0.

• Scoring: strict pass; a checkpoint counts as passed only when the current checkpoint and all black-box tests inherited from previous checkpoints pass.

Inputs/configuration

• xjq: 5 checkpoints, expanding from XPath/XML/HTML/JSON queries to selectors, structured output, and union.

• file_backup: 4 checkpoints, expanding YAML backup scheduling to archive, verify, targets, and incremental state.

• dag_execution: 3 checkpoints covering a task-pipeline DSL, caching, and dynamic cache coverage.

• circuit_eval: 8 checkpoints covering circuit parsing, vectors, three-valued logic, formatting, statistics, equivalence, and optimization.

• code_search: 5 checkpoints covering multilingual text/regex/structured search, AST selectors, and fixes.

• etl_pipeline: 5 checkpoints covering JSON ETL, branching, definitions, parameterized calls, and namespace composition.

• The author publicly released condensed requirements for all 30 checkpoints, the model order, isolated contexts, and the black-box judging mechanism.

Results

• As a tiebreaker, Fable had 16 isolated passes, compared with Sol's 14.

• In the final snapshot, approximately 95% of Sol's code lines triggered at least one slop rule, compared with 86% for Fable and 82%/79% for Kimi K3; the author believes the rules may be too aggressive.

• Sol left 1,318 SLOC in persistent Python test files; the other agents used shell, fixtures, or temporary files, so the figures are not directly comparable.

• The author stresses that these are directional results from one run per model/provider and should not be treated as statistically significant rankings.

Conclusion

On long-horizon coding tasks with progressively disclosed requirements and inherited defects, Sol and Fable had the same strict pass rate, but all models accumulated defects as checkpoints progressed. Sol's code-volume/test-artifact metrics also suggest that “more complete testing” does not equal lower complexity; black-box regression and complexity audits should be run together.

Limitations

• Each model and Kimi provider was run only once, leaving too small a sample for significance inferences.

• Sol, Fable, and Kimi used different harnesses; differences in cost and tooling affect the results.

• The relationship between the slop metric and “code maintainability” has not been validated, and the author also notes that the rules may penalize too heavily.

• The article does not disclose all original repository snapshots for each checkpoint or complete token/latency logs.

Reproduction steps

1. Obtain the SlopCodeBench paper, runner, and problem catalog, and pin the versions of all six challenges.

2. Create a clean context for each checkpoint and expose only the current requirements; retain the code from the previous checkpoint.

3. Pin each model's harness version and tool permissions, and record every code submission, test output, and tool call.

4. Run hidden black-box tests for the current checkpoint and all inherited checkpoints, and score strict passes.

5. Also measure defect accumulation, code growth, duplication, cognitive complexity, and test-file types.

6. Repeat multiple times and cross-run different harnesses before discussing statistical differences.

Original evidence and data

• The article provides the task shapes for 6 challenges, condensed inputs for 30 checkpoints, harness versions, and strict scoring rules.

• The strict pass counts for Sol/Fable/Kimi and some code-quality statistics are disclosed directly, and the author explicitly states the sponsorship and one-run limitations.

Scope of applicability

• Best suited for evaluating long-horizon coding, progressively added requirements, and regression-defect propagation; it is not equivalent to ordinary single-turn code generation.

• The 33.3% tie between Sol and Fable does not imply that they are equivalent in creativity, writing, or knowledge work.

• The “slop” results should be treated only as signals requiring validation, not as direct conclusions about engineering quality.

Source excerpt or observation (short quote for compliance only)

The author characterizes the experiment as “directional, not exhaustive” and emphasizes that all new models continued to accumulate defects.