Devin Team: FrontierCode Extended Benchmark and Long-Horizon Engineering Performance

One-Sentence Takeaway

On the independent FrontierCode Extended benchmark built by the Devin team for real-world software engineering tasks, Kimi K2.7 Code achieved a 39.5% pass rate, placing it firmly in the competitive tier alongside top-tier proprietary models. It excels at generating standalone UI components and self-contained features, but remains constrained by its context window and memory span during long-sequence multi-file refactoring.

Test Environment, Input / Configuration

• Evaluation Benchmark: FrontierCode Extended (a comprehensive benchmark suite by the Devin / Cognition team designed to evaluate real-world end-to-end software engineering tasks) .

• Execution Environment: Real agent execution environments across Devin Desktop and Devin CLI.

• Model Comparison Group: Claude Opus 4.8, GPT-5.5, GLM 5.2, Kimi K2.7 Code.

• Client Constraints: During testing, the Devin client capped the allocated context window for K2.7 / GLM 5.2 at approximately 200K tokens.

Results Data

FrontierCode Extended official published results (Pass Rate) :

Community developer feedback distribution in real-world engineering:

• Self-Contained Features / UI Tasks: Kimi K2.7 generates code exceptionally fast with snappy responsiveness and high out-of-the-box usability.

• **Large-Scale Long-Horizon Refactoring (>15 consecutive files modified) **: Performance degrades when the agent must maintain precise memory of changes in files 1–8 while working through files 9–15.

Conclusion

1. Open-Source Models Enter the Top Tier: Kimi K2.7 Code (39.5%) and GLM 5.2 (43.0%) demonstrated problem-solving capabilities on real-world complex engineering tasks that approach top-tier proprietary models (GPT-5.5 at 44.8%) .

2. Task Suitability Divide:

   • High-Win Scenarios: End-to-end single-feature development, standalone module refactoring, UI/frontend page implementation, and unit test generation.

   • Challenging Scenarios: Long-horizon cascading refactoring involving more than 15 interrelated files and synchronized dependency updates across multiple repositories.

Limitations

• FrontierCode Extended incorporates Devin platform-specific agent toolsets and execution feedback mechanisms; switching to alternative agent frameworks (such as SWE-agent or Aider) may produce different pass rates.

• The client-side limit of a 200K context during testing prevented the model from fully leveraging its native 256K token potential.

Reproduction Steps

1. Select Kimi K2.7 Code as the underlying execution model in Devin Desktop / CLI.

2. Configure two sets of engineering tasks: standalone feature creation and multi-file refactoring.

3. Record the number of execution turns, tool invocation success rate, final build/test pass status, and token consumption.

Raw Evidence and Data

Devin team engineer theodormarcu published the specific percentage scores above in an official announcement, confirming FrontierCode Extended as the comparative benchmark.

Source Excerpts or Observations (Short Compliance Excerpts Only)

• Official Announcement: "GLM 5.2 scores 43.0% and Kimi K2.7 scores 39.5% on FrontierCode Extended — placing them in a competitive tier alongside GPT-5.5 (44.8%) and Claude Opus 4.8 (51.8%) ."

• Senior Community Developer Observation: "200k covers most single-feature work but shows up on large refactors — when you're touching 15 files in sequence and need the model to hold what changed in files 1-8 while working on 9-15, the ceiling matters."