Claude Sonnet 4.6: Computer Use Tool Definitions and Automated Closed-Loop Workflow

One-sentence takeaway

Through standardized computer_20260124, bash_20260124, and text_editor_20260124 tool definitions, combined with an "observe-locate-act-verify" closed-loop system prompt, Claude Sonnet 4.6 can execute desktop and browser cross-application automation with high precision.

Use cases

• Suitable tasks: Operating legacy desktop software without public APIs, filling complex multi-tab web forms, moving data across applications, and cross-system reconciliation.

• Unsuitable tasks: High-frame-rate real-time games, millisecond-level trading, and high-risk operations involving sensitive fund transfers without human review and confirmation.

• Applicable model versions: Claude Sonnet 4.6 (claude-sonnet-4-6).

• Applicable clients, agents, or APIs: Anthropic Messages API (with the anthropic-beta: computer-use-2026-01-24 request header enabled).

• Recommended reasoning tier and parameters: effort=medium is recommended; scale the screen resolution to 1024×768 or 1280×800 to preserve coordinate accuracy and control visual token consumption.

Ready-to-use content

API tool declaration configuration

{
  "model": "claude-sonnet-4-6",
  "max_tokens": 4096,
  "output_config": {
    "effort": "medium"
  },
  "tools": [
    {
      "type": "computer_20260124",
      "name": "computer",
      "display_width_px": 1024,
      "display_height_px": 768,
      "display_number": 1
    },
    {
      "type": "bash_20260124",
      "name": "bash"
    },
    {
      "type": "text_editor_20260124",
      "name": "str_replace_editor"
    }
  ]
}

Companion Computer Use system guidance prompt

<role>
You are an automation agent that operates a computer to accomplish specific business objectives.
</role>

<operating_principles>
1. Observe first: Before clicking or typing, always capture the latest screenshot to confirm the target UI element is visible in the current view.
2. Coordinate calibration: Perform clicks using physical screen coordinates at 1024x768; after clicking, observe screen changes to confirm the action took effect.
3. Fault tolerance and retry: If a click does not trigger the expected response, first check for unclosed modal dialogs or loading overlays, then take a new screenshot and reposition.
4. Keyboard shortcuts: Prefer standard shortcuts (e.g., Ctrl+A / Cmd+A for select all, Ctrl+V for paste) to reduce loss that can occur with character-by-character input.
5. State assertion: After each critical action, confirm the target state through a screenshot or text check before proceeding to the next step.
6. Safety baseline: Before payment confirmation, deleting a production database, or sending bulk email externally, output a prompt requesting operator confirmation.
</operating_principles>

<execution_steps>
1. Parse the initial task and target end state provided by the user.
2. Capture the initial screen image and analyze the current window layout.
3. Plan a subtask path in groups of no more than 10 steps.
4. Loop by calling the computer tool to execute actions (mouse_move, click, type, key, screenshot).
5. When the task ends, output the final execution results and validation evidence.
</execution_steps>

Test/workflow steps

1. Environment setup: Start an isolated Docker container or virtual machine, run an Xvfb virtual desktop, and bind a display port (e.g., :1, resolution 1024x768).

2. API handshake: Send an initial request with the computer-use-2026-01-24 request header, passing the target task and system prompt.

3. Agent loop operation:

   • After the client receives the model's tool_use (e.g., action: "screenshot"), capture an image from the virtual desktop and convert it to base64 PNG.

   • The client returns tool_result to the model.

   • The model returns the next action (e.g., action: "left_click", coordinate: [450, 320]), and the client executes it via xdotool or an OS API.

4. Termination criteria: When the model outputs stop_reason: "end_turn" without calling a tool, check the final state and end the session.

Original evidence and data

• Official documentation states that Sonnet 4.6 significantly reduces "coordinate drift" compared with earlier generations, with first-click hit rate improving by about 22% at 1024×768 resolution.

• Official benchmarks report that on OSWorld-Verified, Sonnet 4.6 reaches a 72.5% task completion rate, nearly matching flagship Opus 4.6 (72.7%), while runtime cost drops by about 40% and response throughput increases by nearly 2×.

• Official guidance recommends combining screenshot calls with local cropping in long-horizon automation to avoid exhausting context quickly from large full-screen visual tokens in a single task.

Scope and limitations

• OS DPI scaling (e.g., 200% Retina displays) causes physical pixels and logical pixels to diverge; coordinate normalization must be handled at the agent client layer.

• Dynamic dropdown menus or hover-triggered overlays can disappear after mouse movement; use keyboard shortcuts or an explicit hover-and-wait strategy.

• For in-browser automation, prefer Playwright/CDP DOM targeting; use pure visual Computer Use only when no DOM interface is available or when crossing native desktop applications.

Source excerpt or observation (short compliant paraphrase only)

Anthropic states: "Computer use allows Claude to interact directly with standard user interfaces, filling the gap where dedicated APIs are unavailable." (paraphrase from official technical documentation)