In ITBench-AA — the real-world enterprise-grade SRE operations benchmark jointly launched by IBM Research and Artificial Analysis — Qwen3.7-Max debuted at #3 globally upon release, demonstrating outstanding cross-system root-cause localization capabilities during multidimensional snapshot analysis of complex sandboxed Kubernetes incidents.
Evaluation benchmark: ITBench-AA (independent reproduction edition of IBM ITBench) .
Test scenarios: 59 real-world enterprise-grade Kubernetes troubleshooting tasks (40 public scenarios + 19 unreleased private validation scenarios) , with each task tested 3 times repeatedly to eliminate random error.
Input data payload: Offline incident snapshots (including full Prometheus metrics, OpenTelemetry Traces, K8s Events, system alert logs, and application service topology graphs) .
Execution framework: Stirrup (open-source Agent execution sandbox, granting models autonomous Shell interaction and file retrieval permissions) .
Scoring criterion: Average Precision at Full Recall (calculating TP / (TP + FP) under the premise of zero false negatives) .
Diagnostic output: The Agent is required to complete trace troubleshooting inside the sandbox and output a standardized JSON incident diagnostic report explicitly specifying root-cause entities (Deployment, Service, Pod, Namespace, NetworkPolicy, ConfigMap, etc.) .
Thinking mode: Thinking mode enabled, granting tool exploration and log filtering capabilities.
| Model | Reasoning tier | ITBench-AA Composite Precision | Average Interaction Turns (Turns) | Single-Task Inference Latency (Min) |
|---|---|---|---|---|
| GPT-5.6 Sol | max | 56.2% | 30.8 | 5.18 |
| GPT-5.6 Terra | max | 51.0% | 37.7 | 3.74 |
| Qwen3.7-Max | reasoning | **Debut Top 3 (#3) ** | ~35-40 | ~3.5-4.0 |
| Kimi K3 | max | 47.7% | 39.0 | 13.37 |
| Claude Opus 4.7 | max | 46.7% | 68.2 | 10.16 |
| GPT-5.5 | xhigh | 45.8% | 30.9 | 3.55 |
| GPT-5.6 Luna | max | 40.3% | 45.9 | 3.77 |
| Claude 4.5 Haiku | reasoning | 27.3% | 40.6 | 2.65 |
| gpt-oss-120b | high | 5.6% | 74.3 | 3.10 |
| Nemotron 3 Super | default | 1.1% | 98.8 | 5.61 |
Case study: Feature Flag configuration triggers downstream service avalanche:
Failure symptom: Sudden CPU spike and latency breach in the advertising service under the otel-demo namespace.
Surface-level trigger: Downstream Ad Deployment Pod triggered high-load alerts.
Model performance: Qwen3.7-Max successfully traced upstream along the topology relationship, accurately pinpointing the abnormal flag in the flagd-config ConfigMap ( adHighCpu: true ) rather than merely flagging the affected downstream Pods, avoiding false attribution.
Case study: Environment variable port mismatch leads to communication breakdown:
Failure symptom: Shipping service failed to call Quote service.
Model performance: Through Trace analysis and Deployment environment variable comparison, the model accurately captured the configuration defect where QUOTE_ADDR was mistakenly written with an invalid port ( quote:0000 ) .
First-tier performance in enterprise IT and SRE scenarios: When facing hundreds of megabytes of multidimensional mixed inputs spanning metrics, logs, and trace topologies, Qwen3.7-Max effectively utilizes Shell tools for targeted grep and correlation analysis, with its overall troubleshooting efficiency and localization precision firmly placed among the global leaders (#3) .
Balance between interaction turns and latency: An average of 35–40 interaction turns and approximately 3.5 minutes of decoding latency makes it more concise and efficient than Claude Opus 4.7 (68.2 turns, 10.16 minutes) .
Even for the top three models, overall precision on ITBench-AA has not yet broken through 60%, indicating that fully autonomous enterprise-grade SRE operations remain in a high-difficulty frontier stage; AI is currently suited only as an auxiliary troubleshooting Copilot and should not be directly granted unreviewed self-healing write permissions on production clusters.
The evaluation is based on static offline incident snapshots and does not cover dynamic real-time fault injection or long-cycle closed-loop self-healing interactions.
Clone the official evaluation suite github.com/ArtificialAnalysis/ITBench-AA and the execution engine Stirrup.
Configure the sandbox environment to mount the Kubernetes incident snapshot data package provided by IBM.
Point the endpoint to qwen3.7-max, run the complete 59-item SRE evaluation pipeline, and compare the generated JSON diagnostic reports against the Ground Truth.
IBM Research and Artificial Analysis emphasize: “ITBench-AA tests AI agents on Kubernetes incident root-cause analysis... Qwen3.7-Max just hit #3 on ITbench-AA, demonstrating how well models handle real-world enterprise IT tasks, agentic-style”.
Qwen3.7 Max
