Under a single-turn code-generation protocol with no agent, tools, scaffold, or sampling and with temperature=0, the experimental repository reports that Ox Alpha scored 49/175, or Pass@1=28.0%, on LiveCodeBench release_v6, with lower pass rates as difficulty increased.
Entry point: OpenRouter stealth/ox-alpha, the free preview available on 2026-08-21.
Dataset: LiveCodeBench code_generation_lite test6.jsonl, release_v6, with 175 problems.
Decoding: Greedy, temperature=0, one attempt per problem.
Model input: Problem statement + starter code, with an instruction to return a Python code block.
Execution: A new subprocess for each test case; stdin/stdout comparison with JSON normalization; function problems used argument parsing and a function-call wrapper.
Hardware: i5 / 8 GB RAM / no GPU; inference was 100% remote.
Reliability: API errors used exponential backoff, and each problem had a checkpoint so the run could resume.
| Difficulty | Passed | Problems | Pass@1 |
|---|---|---|---|
| Easy | 22 | 43 | 51.2% |
| Medium | 16 | 52 | 30.8% |
| Hard | 11 | 80 | 13.8% |
| Overall | 49 | 175 | 28.0% |
Generation failures: 0; all 175 attempts produced executable code.
Online raw report: https://xnasarx.github.io/ox-alpha-benchmarks/results/report.html
Reproducible experiment repository: https://github.com/xnasarx/ox-alpha-benchmarks
Raw per-problem results: results/lcb/latest.json in the repository.
Clone the repository and pin the commit; inspect README.md, scripts/, oxbench/, and results/.
Obtain the same LiveCodeBench release_v6 data and fix the problem order, prompt wrapper, temperature=0, and one attempt per problem.
Request stealth/ox-alpha through OpenRouter, saving the model ID, provider, request time, response, errors, and checkpoint.
Run the repository evaluator locally against the hidden tests, outputting per-problem passed/failed status, difficulty aggregates, and generation-failure count.
When comparing with a known-version model, use the same problem set, wrapper, timeout, decoding, and execution environment.
This is a more reproducible raw capability baseline than simply observing that the model “looks able to code”: it supports a 28.0% single-turn coding result for Ox Alpha under this protocol and shows a 13.8% pass rate on Hard problems. It does not represent real-world development ability with tools, long context, iterative repair, or an agent harness.
This is one single-turn greedy result; it cannot estimate Pass@k or the sampling distribution.
The repository displays DeepSeek vendor results alongside it, but those figures do not use the same experimental protocol and cannot form a strict ranking.
The 20-second test timeout, temperature, and no-scaffold setup are suitable for reproducing the raw baseline, not necessarily the best-use configuration.
Ox Alpha
