Qwen3.7-Max: AA-Omniscience Knowledge Reliability and Hallucination Rate Benchmark

One-sentence takeaway

Independent evaluation on the AA-Omniscience benchmark indicates that Qwen3.7-Max demonstrates superior uncertainty calibration: rather than blindly outputting incorrect answers with unearned confidence, it is more inclined to acknowledge its knowledge boundaries, outperforming open-source LLMs in the same tier on pure-text knowledge reliability and anti-hallucination metrics.

Test environment

• Evaluation framework: Artificial Analysis AA-Omniscience (comprehensive domain knowledge and hallucination benchmark suite) .

• Evaluation logic:

  • **Omniscience Index (Omniscience Index) **: Rewards correct answers and penalizes hallucinated errors; explicit refusal when uncertain ( "I don't know" ) is not penalized. Scores range from -100 to 100 points.

  • **Hallucination Rate (Hallucination Rate) **: The proportion of incorrect answers among all samples that were not answered completely correctly ( Incorrect / (Incorrect + Partial + Not Attempted) ) . Lower is better.

  • **Omniscience Accuracy (Omniscience Accuracy) **: The proportion of completely correct answers across all questions.

• Hardware and endpoints: Executed directly by Artificial Analysis via official model APIs on dedicated independent benchmark hardware.

Inputs/configuration

• Input format: Complex knowledge question-answering spanning broad domains including science, history, technology, medicine, law, and multilingual subjects.

• Model configuration: Standard knowledge Q&A prompt without injecting additional misleading or deceptive conditions, assessing the model's native knowledge and uncertainty confidence calibration.

Results data

1. Cross-model AA-Omniscience core metric comparison

2. Key insights on hallucination rate and refusal mechanisms

• Divergence under penalty mechanisms: Several models exhibit seemingly decent surface-level accuracy (e.g., 45%–49%) , yet fabricate facts with high confidence when encountering unknown knowledge (e.g., some models exhibiting an error rate exceeding 50% on non-correct answers) , causing their Omniscience Index to hover near zero or drop into negative territory.

• Confidence behavior of Qwen3.7-Max: When facing fine-grained domain knowledge without definitive certainty, Qwen3.7-Max more consistently activates conservative refusal strategies, effectively curbing the critical flaw of “confidently lying.”

Conclusions

• In scenarios with stringent factual accuracy requirements, such as Retrieval-Augmented Generation (RAG) and compliance review, Qwen3.7-Max demonstrates well-calibrated confidence estimation.

• Compared with earlier versions and other open-source derivative models that recorded negative indices on Omniscience, Qwen3.7-Max significantly mitigates factual drift across multi-turn interactions.

Limitations

• While a conservative refusal strategy substantially lowers the fatal hallucination rate, it can occasionally lead to excessive caution under ambiguous user prompts with incomplete information, necessitating tailored system prompts to guide reasonable deduction.

• Pure knowledge benchmarks do not equate to logical correctness in code generation or tool calling.

Reproduction steps

1. Construct a QA dataset containing 100 questions with traps or long-tail common knowledge (with 30% intentionally configured with unknowable premises) .

2. Query qwen3.7-max under identical temperature (temperature=0.1) settings, recording the ratio of directly fabricating false answers versus identifying false premises / acknowledging unknown facts.

3. Compute the composite ROC/AUC confidence curve.

Source excerpt or observation (brief excerpt for compliance only)

A tech community discussion on Hacker News noted: “The non-hallucination rate in AA-omniscience is SOTA... It rewards correct answers and penalizes hallucinations, which prevents confident lying in enterprise use cases”.