Google demonstrates Argon workflows for robot control, Real2Sim, physical understanding, and self-improvement through observation, execution, verification, and correction—not a general hardware safety protocol.
Suitable tasks: Mobile manipulation, zero-shot robotic-arm control, video world-model steering, robot or human video to MuJoCo simulation, sketch to simulation, hindsight video labeling, value-function estimation, simulation data generation, and hardware self-improvement.
Unsuitable tasks: Treating the examples as safe control programs for arbitrary robots or hardware; the article does not provide a unified hardware adapter, permission model, emergency-stop procedure, or deployment acceptance checklist.
Applicable model version: Gemini 4 Argon; the Gemini API model ID is gemini-4-argon.
Applicable clients, agents, or APIs: The Gemini API Interactions API (Google says the Google GenAI SDKs support it from day one); Google AI Studio (for Google AI Ultra subscribers); and Google Antigravity (for Google AI Ultra subscribers, for complex planning, /goal runs, and multi-agent orchestration).
Recommended reasoning level and parameters: Not specified. The article does not disclose temperature, a reasoning level, a context limit, tool schemas, or unified sampling settings.
The article publishes multiple task prompts, but they depend on specific images, videos, observation interfaces, simulators, and robot tools. The following preserves one copyable zero-shot-control prompt from the source; users must provide the input files and local services themselves.
[Attached: wrist_cam.png, obs.json]
Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.
## Execution Discipline
- Simply output 1 sentence observation followed immediately by your `curl` command (with `# Plan:` on Turn 1 and `# Thought:` on every turn). Everything needed is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`.
## Visual Coordinate Reference (Annotated on wrist_cam.png)
- `+X` (UP in image): Away from robot base | `-X` (DOWN): Towards robot base
- `+Y` (LEFT in image): World left | `-Y` (RIGHT): World right
- `+Z`: Height above table | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`)
- `Cyan + crosshair`: Current 3D pinch center
- `Cyan [ + ] brackets`: Current jaw opening and closing line
- `Green line`: Current object long-axis orientation (`yaw_deg`: 0° = vertical along +X, 90° = horizontal along +Y)
- `pitch_deg = 90°`: Straight down (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial approach along finger axis
## Closed-Loop Turn Workflow
1. Inspect current state & view `wrist_cam.png`:
`curl -s http://127.0.0.1:8081/obs` (returns `ee_pos`, `holding`, `status`, `workspace`, `wrist_cam_path`).
2. Select VLM call granularity upfront on Turn 1:
- Coarse (2–4 calls): `look_at` -> `grasp` -> `place` -> `done`
- Pre-Grasp Hover Verify (4–7 calls): `look_at` -> `goto(z=0.04)` (verify cyan `[ + ]` brackets straddle target) -> `grasp` -> `place_until_contact` -> `done`
- Fine Visual Servoing (8–20 calls): `look_at` -> `grasp` -> pre-pose `goto` -> closed-loop `move_by` / `move_until_contact` -> `done`
3. Send motion primitives (`POST /plan`) and verify visually after every turn in the updated `wrist_cam.png`. If `status == "grasp_failed"`, call `robot.look_at(...)` to re-observe and recover closed-loop.
## Code-as-Policies Primitives (`POST http://127.0.0.1:8081/plan`)
robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015)
x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015)
robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90)
robot.place(x=0.24, y=0.06, z=0.075)
robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002)
robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10)
robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04)
robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002)
robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()This is not a general-purpose robot SDK document: wrist_cam.png, obs.json, /obs, /plan, and robot.* are environment conventions from the article's example.
| Workflow | Inputs | Methods and steps | Outputs or acceptance signals |
|---|---|---|---|
| Spot mobile manipulation | Two Diet Coke cans, Spot views, and the spot-mobile-manipulation skill | First build and practice with an articulated MuJoCo digital twin of the foot-pedal bin; then pick up the cans, inspect the wrist camera, maintain three-leg tripod balance, and step on the pedal; before release, foot-verifier and hand-verifier agents separately verify the foot target and 3D hand-to-bin alignment | The cans enter the black foot-pedal bin; the article requires both verifications before release |
| Zero-shot control | wrist_cam.png, obs.json | Inspect /obs; choose coarse (2–4 calls), hover verification (4–7 calls), or fine visual servoing (8–20 calls) on the first turn; send actions through POST /plan; inspect the updated wrist image after every turn; call look_at again after grasp_failed | The mug is upright and the crumpled napkin is inside it; each turn includes visual verification |
| Omni control | Intermediate frames from the latest rollout and the task_goal | In a Gemini Omni visual loop, write timestamped action prompts one turn at a time; take one simple single-direction action at a time, retract inactive hands, state which workspace areas become empty after an object moves, and wrap the output in <omni_prompt> | After inspecting intermediate keyframes and refining prompts, arrange five Scrabble letters as A-R-G-O-N; the article's figure caption calls this a 33-step rollout |
Robot view to simulation: Input [Attached: robot_rollout_episode]. Replay the recorded observation.state with joint-space PD; iteratively optimize visual matching for the three cameras (head_left, left_wrist, right_wrist) and match the timing, magnitude, and direction of contact force/torque at each Sharpa fingertip in the sensor frame to tactile_force. The outputs are a MuJoCo scene, rollout results, and an interactive HTML session viewer: a synced timeline at the bottom, a Blender 3D scene on the left, and real/sim camera and finger-force visualizations on the right.
Human view to simulation: Inputs egocentric_human_video.mp4, candidate_sim_views.png, and hand_3d_keypoints.json. Identify relevant objects and replace crude boxes with compound convex-decomposed meshes; retarget human wrist and finger keypoints to a 7-DoF arm and 22-DoF hand; run MuJoCo at 500 Hz (dt = 0.002s); and verify opposing thumb-and-finger force closure, a lift above 2 cm, and stable placement. The outputs are scene.xml and a Blender studio render.
Sketch to simulation: Input hand_drawn_throw_sketch.jpg. The article's example builds an Apollo 2 humanoid throwing scene in mujoco_warp, synthesizes a 76-DoF whole-body trajectory, runs a four-phase throw at 50 Hz for 140 steps and 2.80 s, verifies that 6/6 target boxes fall in closed-loop physics and that pelvis drift stays below 3 cm, and exports a Blender 4.5 Cycles render.
Hindsight labeling: Input the 136-second ALOHA 2 origami-fox video Aloha2_Origami_20250309.mp4; output four-level, time-aligned JSON: the G1 episode goal, 6 contiguous G2 semantic subtasks, 17 G3 contact-level atomic primitives, and QC reweighting. For non-progressing segments such as slips, pauses, and retries, keep task_progress flat and downweight it by 0.15x–0.25x.
Odd-ball puzzle: Input stereo RGB views from dual-arm Franka FR3 and proprioception (add_privileged_task_observations=False). Rely only on camera segmentation, triangulation, and visual scale tilt (left heavy +1, right heavy -1, balanced 0) to put the odd ball into the Heavy (red) or Light (blue) tray within no more than 3 weighings.
Value function: Inputs umi_head.mp4, umi_left_gripper.mp4, and umi_right_gripper.mp4; generate positive, neutral, and negative progress slopes around contact transitions, identify important segments, and divide the trajectory into 28 segments.
Simulation data for action improvement: Input one real_tabletop_photo.jpg. Reconstruct procedural meshes and MuJoCo collision bodies for 9 party props; define a 77-task single-arm/bimanual curriculum; synthesize closed-loop IK trajectories with ±10 cm pose randomization; verify reward == 1.0; and export 4,500+ simulation episodes. The article reports that real-robot Success Rate rose from 56% to 75% and completion time fell from 35 s to 24 s; these are the page's case-study results, not an independent replication.
Hardware self-improvement: The article gives three stages. Stage 1 only measures objects, generates a dry-run plan, annotates the overhead image, and explicitly says not to move the hardware; Stage 2 moves the gripper above the brick without touching it, verifies with the wrist camera, and records execution from both cameras; Stage 3 performs pick-and-place, reviews logs and video after failures, diagnoses the root cause, checks the fix in a dry run, and then optimizes speed within safety limits. After success, the target is at least 2× faster unless the user authorizes more or provides feedback.
Paper reproduction: Input paper.pdf, with optional author_checkpoints/ or urdf_assets/. Reproduce the controller in MuJoCo WebAssembly using the paper's physics timestep, actuator gains, torque limits, and kinematic constraints; generate an index.html simulator and a scrollytelling.html visual essay; and use headless Chromium to assert 0 console errors and non-zero task progress over 300+ frames. The article says this method has covered 68 interactive browser sandboxes, but provides no per-reproduction data.
The article says Argon scored 87% on the Embodied Reasoning (ERQA) benchmark; the page does not publish the full task set, run configuration, evaluator, or an independent replication.
The article describes Argon as a general-purpose physical-reasoning model spanning the robotics lifecycle and organizes examples into Robot control, Real2Sim, Physical understanding, and Research and self-improvement.
The model and entry-point configuration given by the page is: use gemini-4-argon with the Gemini API Interactions API; Google AI Studio and Google Antigravity are available to Google AI Ultra subscribers; no unified API request body is provided.
Observable outputs in the page include MuJoCo scenes, scene.xml, Blender/Cycles renders, interactive HTML viewers, time-aligned JSON, video-world-model action prompts, closed-loop action primitives, and simulation episodes; the exact output depends on the task.
The article is Google's capability showcase and collection of representative prompts, not an independent robotics safety certification; 87%, 56%→75%, and 35 s→24 s should all be retained as page-reported figures.
Real-robot control depends on specific hardware, camera calibration, skills, action services, and local interfaces. The article does not provide a general permission model, emergency-stop procedure, collision-safety threshold, or deployment checklist.
The Real2Sim example explicitly prohibits “cheating”: after t = 0, do not change MjModel parameters; apply external forces or torques to objects; use welds or equality constraints; use feedforward torques beyond joint-space PD; set or roll back object qpos/qvel; disable or exclude collisions; or report contact forces that did not come from the MuJoCo solver (for example, by copying logged tactile readings). Task objects may move only through real MuJoCo contact (mj_step) with the robot hands and the environment.
The robot-operation examples require observation and verification after key actions, but that does not make them directly executable on real hardware; they still require separate validation on the target hardware, in simulation, and in a controlled environment before deployment.
The phrase “subject to safety rules from AGENTS.md” appears only in the self-improvement example; the article does not provide those rules, so no safety policy should be inferred from it.
Confirm the entry point and permissions first: the paid Gemini API tier, Google AI Studio for Google AI Ultra, or Antigravity; fix the model ID at gemini-4-argon, and record the actual reasoning and sampling settings that the article leaves unspecified.
Prepare the required input files, camera calibration, observation interface, simulator, and action service for the target task; do not treat the examples' /obs, /plan, or robot.* as general APIs.
Reproduce the workflow in simulation or without hardware motion first: inspect the observation, goal, coordinate system, and action plan, then execute turn by turn and verify the updated image or keyframes.
For Real2Sim, record visual matching, contact-force/torque matching, and every prohibited cheating path; for self-improvement, record logs, two-camera video, failure causes, dry runs, and human authorization.
Report inputs, actions, outputs, failures, human interventions, latency, and cost separately. Do not treat the page's benchmark or case-study figures as your own replication results.
Google Developers Blog: https://developers.googleblog.com/gemini-4-argon-for-robotics/
Gemini 4 Argon