Qwen3.7-Max ITBench-AA Enterprise IT Operations and SRE Root-Cause Analysis Benchmark

One-sentence takeaway

In ITBench-AA — the real-world enterprise-grade SRE operations benchmark jointly launched by IBM Research and Artificial Analysis — Qwen3.7-Max debuted at #3 globally upon release, demonstrating outstanding cross-system root-cause localization capabilities during multidimensional snapshot analysis of complex sandboxed Kubernetes incidents.

Test environment

• Evaluation benchmark: ITBench-AA (independent reproduction edition of IBM ITBench) .

• Test scenarios: 59 real-world enterprise-grade Kubernetes troubleshooting tasks (40 public scenarios + 19 unreleased private validation scenarios) , with each task tested 3 times repeatedly to eliminate random error.

• Input data payload: Offline incident snapshots (including full Prometheus metrics, OpenTelemetry Traces, K8s Events, system alert logs, and application service topology graphs) .

• Execution framework: Stirrup (open-source Agent execution sandbox, granting models autonomous Shell interaction and file retrieval permissions) .

• Scoring criterion: Average Precision at Full Recall (calculating TP / (TP + FP) under the premise of zero false negatives) .

Inputs/configuration

• Diagnostic output: The Agent is required to complete trace troubleshooting inside the sandbox and output a standardized JSON incident diagnostic report explicitly specifying root-cause entities (Deployment, Service, Pod, Namespace, NetworkPolicy, ConfigMap, etc.) .

• Thinking mode: Thinking mode enabled, granting tool exploration and log filtering capabilities.

Results data

1. ITBench-AA Core Rankings and Accuracy Comparison (SRE Tasks)

2. Classic Incident Diagnosis Case Studies

• Case study: Feature Flag configuration triggers downstream service avalanche:

  • Failure symptom: Sudden CPU spike and latency breach in the advertising service under the otel-demo namespace.

  • Surface-level trigger: Downstream Ad Deployment Pod triggered high-load alerts.

  • Model performance: Qwen3.7-Max successfully traced upstream along the topology relationship, accurately pinpointing the abnormal flag in the flagd-config ConfigMap ( adHighCpu: true ) rather than merely flagging the affected downstream Pods, avoiding false attribution.

• Case study: Environment variable port mismatch leads to communication breakdown:

  • Failure symptom: Shipping service failed to call Quote service.

  • Model performance: Through Trace analysis and Deployment environment variable comparison, the model accurately captured the configuration defect where QUOTE_ADDR was mistakenly written with an invalid port ( quote:0000 ) .

Conclusions

• First-tier performance in enterprise IT and SRE scenarios: When facing hundreds of megabytes of multidimensional mixed inputs spanning metrics, logs, and trace topologies, Qwen3.7-Max effectively utilizes Shell tools for targeted grep and correlation analysis, with its overall troubleshooting efficiency and localization precision firmly placed among the global leaders (#3) .

• Balance between interaction turns and latency: An average of 35–40 interaction turns and approximately 3.5 minutes of decoding latency makes it more concise and efficient than Claude Opus 4.7 (68.2 turns, 10.16 minutes) .

Limitations

• Even for the top three models, overall precision on ITBench-AA has not yet broken through 60%, indicating that fully autonomous enterprise-grade SRE operations remain in a high-difficulty frontier stage; AI is currently suited only as an auxiliary troubleshooting Copilot and should not be directly granted unreviewed self-healing write permissions on production clusters.

• The evaluation is based on static offline incident snapshots and does not cover dynamic real-time fault injection or long-cycle closed-loop self-healing interactions.

Reproduction steps

1. Clone the official evaluation suite github.com/ArtificialAnalysis/ITBench-AA and the execution engine Stirrup.

2. Configure the sandbox environment to mount the Kubernetes incident snapshot data package provided by IBM.

3. Point the endpoint to qwen3.7-max, run the complete 59-item SRE evaluation pipeline, and compare the generated JSON diagnostic reports against the Ground Truth.

Source excerpt or observation (brief excerpt for compliance only)

IBM Research and Artificial Analysis emphasize: “ITBench-AA tests AI agents on Kubernetes incident root-cause analysis... Qwen3.7-Max just hit #3 on ITbench-AA, demonstrating how well models handle real-world enterprise IT tasks, agentic-style”.