Independent LiveCodeBench v6: Ox Alpha's Raw Pass@1

One-sentence takeaway

Under a single-turn code-generation protocol with no agent, tools, scaffold, or sampling and with temperature=0, the experimental repository reports that Ox Alpha scored 49/175, or Pass@1=28.0%, on LiveCodeBench release_v6, with lower pass rates as difficulty increased.

Test environment

• Entry point: OpenRouter stealth/ox-alpha, the free preview available on 2026-08-21.

• Dataset: LiveCodeBench code_generation_lite test6.jsonl, release_v6, with 175 problems.

• Decoding: Greedy, temperature=0, one attempt per problem.

• Model input: Problem statement + starter code, with an instruction to return a Python code block.

• Execution: A new subprocess for each test case; stdin/stdout comparison with JSON normalization; function problems used argument parsing and a function-call wrapper.

• Hardware: i5 / 8 GB RAM / no GPU; inference was 100% remote.

• Reliability: API errors used exponential backoff, and each problem had a checkpoint so the run could resume.

Raw results

• Generation failures: 0; all 175 attempts produced executable code.

• Online raw report: https://xnasarx.github.io/ox-alpha-benchmarks/results/report.html

• Reproducible experiment repository: https://github.com/xnasarx/ox-alpha-benchmarks

• Raw per-problem results: results/lcb/latest.json in the repository.

Reproduction steps

1. Clone the repository and pin the commit; inspect README.md, scripts/, oxbench/, and results/.

2. Obtain the same LiveCodeBench release_v6 data and fix the problem order, prompt wrapper, temperature=0, and one attempt per problem.

3. Request stealth/ox-alpha through OpenRouter, saving the model ID, provider, request time, response, errors, and checkpoint.

4. Run the repository evaluator locally against the hidden tests, outputting per-problem passed/failed status, difficulty aggregates, and generation-failure count.

5. When comparing with a known-version model, use the same problem set, wrapper, timeout, decoding, and execution environment.

Conclusion and applicability boundary

This is a more reproducible raw capability baseline than simply observing that the model “looks able to code”: it supports a 28.0% single-turn coding result for Ox Alpha under this protocol and shows a 13.8% pass rate on Hard problems. It does not represent real-world development ability with tools, long context, iterative repair, or an agent harness.

Limitations

• This is one single-turn greedy result; it cannot estimate Pass@k or the sampling distribution.

• The repository displays DeepSeek vendor results alongside it, but those figures do not use the same experimental protocol and cannot form a strict ranking.

• The 20-second test timeout, temperature, and no-scaffold setup are suitable for reproducing the raw baseline, not necessarily the best-use configuration.