DeepSeek-V4-Pro XSCT Bench Two-Case Comparison: Strong Planning, Weak Clarification

One-sentence takeaway

A two-testcase comparison on the same platform with the same model under test shows that V4-Pro is close to a perfect score on "autonomous planning and execution" (自主规划执行) (base 98.0 / advanced 92.6), yet manages only 68.5 on the persona testcase for "ambiguous-requirement clarification" (模糊需求澄清) — structured tool planning is a strength; persona-driven dialogue that must proactively probe ambiguous requirements is a weak spot.

Use cases

• Suitable tasks: Judging whether V4-Pro fits "agent tool tasks that plan first and then execute" (evidence: strong) and "customer-service/assistant-style role-play dialogue that needs to clarify ambiguity" (evidence: weak); serving as a structural reference for writing agent system prompts.

• Unsuitable tasks: Treating the two testcase scores as an overall capability ranking; treating LLM-as-a-Judge comments as human review conclusions.

• Applicable model versions: deepseek-v4-pro (the platform's model name; snapshot and effort are not disclosed — see boundaries).

• Applicable clients, Agents, or APIs: The platform's unified harness; reproduction on the user side requires your own tool-calling environment.

• Recommended reasoning levels and parameters: Not disclosed; when reproducing, you must record and fix effort/temperature yourself, otherwise cross-comparison is impossible.

Test environment

• Platform and method: XSCT Bench — LLM-as-a-Judge with multiple judges (CLAUDE/GEMINI/KIMI), evidence anchoring, difficulty tiers, and scoring separated from the model under test; the methodology page states that it "lacks Ground Truth validation" and that "testcase coverage has blind spots".

• Testcase 1 (l_agent_008, autonomous planning and execution): Agent MCP-type text generation. The system prompt requires leading with <plan>, JSON tool calls, reviewing in <observation>, and closing with <summary>; the user prompt asks the model to continue a multi-step task that involves reading the README, checking config/, and not reading secrets.env. Score: base 98.0 / advanced 92.6, both passed; the advanced item requires fault tolerance for read_file failures and compiling a list of failures.

• Testcase 2 (l_alice_persona_002, ambiguous-requirement clarification): Persona-driven conversational text generation. The system prompt is the complete persona of Alice, a 26-year-old remote assistant (addressing rules, speaking style, don't guess / don't act rashly, WLB boundaries); the user task contains a double ambiguity around "organizing files" (文件整理). Score: overall 68.5 / base 68.5 — passed, but the review identified core defects.

• Review summary (testcase 2): CLAUDE — "only asked where the file was, completely missing the ambiguity in the 'organize' action itself"; GEMINI — excellent adherence to persona and taboos, but incomplete clarification; KIMI — "appears to have asked, but didn't get to the point".

• Additional context: On the platform leaderboard, deepseek-v4-pro scores 87.7 overall (base 89.1 / advanced 87.4 / hard 86.5), tied with kimi-for-coding and GLM-5v-turbo in that tier.

Input/configuration

Both testcase pages disclose the complete system prompt, user prompt, task requirements (scoring anchors), the model's generated output, the multi-judge review comments, and the base/advanced scores. Not disclosed: the tested model snapshot, effort level, temperature, repeat count, and the calling harness.

Results data

Conclusion

V4-Pro is strong at structured tool planning and execution, but weak at persona-driven dialogue that requires proactively clarifying ambiguous requirements; the single-platform, two-testcase evidence is directional, not a general ranking.

Limitations and reproduction steps

• Limitations: The tested model's snapshot, effort level, temperature, repeat count, and calling harness are not disclosed, and scores change as the platform updates, so reproduction must first lock these variables. The evidence comes from only one platform and two testcases. LLM-as-a-Judge comments may carry systematic bias relative to human judgment (a preference for format completeness and normative tone).

• Reproduction steps: Rerun the two testcases with your own tool-calling environment while locking effort and temperature (and recording the snapshot) so results can be compared horizontally; treat the page as collected as authoritative, since the platform updates results (the "96" and "83" figures in search summaries differ from the current page values of 98.0/68.5). Single-case scores cannot be generalized into an overall capability ranking.

Original evidence and data

• Testcase 1 — actual model output (visible at collection): The <plan> declares safety constraints (not reading secrets.env); the model then runs read_file on the README and list_directory on config/ in sequence; on the advanced item it fully completed the "skip on failure + record the reason + compile and write the summary" flow, and the review called it a "high-quality agent execution example".

• Testcase 2 — full actual model output (visible at collection): "Boss, I don't see that file you mentioned on my end — could you send it over? Also, what should I call you?" — it clarified only that the file was missing, and never asked about the ambiguity of "organize".

• Both testcase pages disclose: the complete system prompt, user prompt, task requirements (scoring anchors), the generated output, the multi-judge review comments, and the base/advanced scores.

• The "96" and "83" scores in the search summary do not match the current page values (98.0/68.5), indicating that the platform updates its results; treat the page as collected as authoritative.

Judge review quotes (testcase 2): CLAUDE — "only asked where the file was, completely missing the ambiguity in the 'organize' action itself"; GEMINI — persona and taboo adherence were excellent, but clarification was incomplete; KIMI — "appears to have asked, but didn't get to the point".

Applicability boundaries

• The tested model's snapshot, effort level, temperature, repeat count, and calling harness are not disclosed; scores change as the platform updates, and reproduction requires locking these variables first.

• The evidence from one platform and two testcases is limited in strength: it can serve as directional evidence for "strong planning, weak clarification", but cannot replace official benchmarks or self-built evaluations.

• Testcase 1 has low personalization and heavily constrained output formatting, so high scores come easily; testcase 2 emphasizes persona tone and proactive clarification, exposing a "prompt requirements pulled off track by format" problem rather than pure reasoning ability.

• LLM-as-a-Judge comments may differ systematically from human judgment (a preference for format completeness and normative tone).

Source excerpt or observation (compliance short quote only)

Testcase 2 KIMI review, verbatim: "The candidate output seriously lost focus on the core task: it handled only a single ambiguity, and did so inappropriately (assuming the file was not received), completely missing the action ambiguity of 'organize', violating the principle of 'don't guess, don't act rashly — ask the most critical question first'."