I Benchmarked GPT-5.6 Sol/Luna/Terra by Role: Role-Based Evaluation

Summary

The author did not collapse the models into a single overall score, instead measuring them in roles such as strategic decision-making, repository execution, and code repair. The results show that Sol high is suited to the main strategic session, while Sol medium is faster and uses fewer resources; in this small-sample execution test, Luna high/max did not beat the existing routing and even showed signs of over-reasoning.

Results

• In strategic tasks, Luna max scored 88, tying Terra max, but below Sol high at 94 and Sol max at 90.

• Repository execution brief: Sol high scored 93 in 80.73 seconds with 1,818 reasoning tokens; Sol medium scored 91 in 70.66 seconds with 779 tokens.

• In the small repair, Luna high went 4/4 in 52.59 seconds; Luna max went 4/4 in 121.34 seconds, which the author described as “heavily over-reasoned.”

• The final routing did not give Luna a fixed slot; the author excluded Luna from the main-session, review, and implementation-rollback paths.

• The author explicitly noted that the sample was small, the evaluator knew the model identities, CLI latency was mixed in, and subscription consumption was not equivalent to API token cost; the results cannot be generalized.

Original article

I ran a small role-based benchmark: three strategic decision memos, one repository-grounded execution brief, and two pla… This is a necessary excerpt; read the original source for full context.