Hydra-0 Treats Robot Actions as Pixel Motion
A team from NVIDIA, Brown, Columbia, and Harvard posted Hydra-0 on Aug. 18: a world model that conditions on action flow, meaning robot actions as pixel motion in the camera plane instead of joint commands.
Their best setup reports 90.4% lower robot-motion error and 60.2% lower object-motion error than an action-conditioned Cosmos 2.5 baseline. On the RoboLab benchmark, replayed versus reference success rates hit Pearson r = 0.96.
Why pixel tracks instead of joints
Joint commands encode one robot’s kinematics. The same end-effector command can look different on another arm. Hydra-0 maps a candidate command through the controller and physics in Isaac Lab, then projects visible robot-surface points into the image. The video model sees where the robot should move on screen.
At training time they recover the same tracks from videos when calibration is missing: dense flow plus grounded masks. The training corpus is 2,201.7 hours after filtering, spanning DROID, ABC-130k, MolmoAct2, EgoDex, Deform360, and smaller cloth-fold sets, including a Unitree H1 fold-clothes slice.
They plug the same interface into Cosmos 2.5 and Wan2.2. A four-step distilled Wan2.2 A14B checkpoint is the strongest row in their table. Few-step distillation reaches 62.0 frames per second generation-only on one H100, a 16.0× speedup versus the bidirectional teacher, excluding VAE decode.
Inverse mode: object flow in, robot motion out
Flip the interface and the model becomes a world action model. Give it desired object flow from a human demo, withhold gripper flow, and a trained head maps latents to executable actions. They show a flexible-pipe-bending task on a real bimanual setup. No task-specific expert robot demonstrations, they say. The paper also notes roughly 1 cm of grasp imprecision as a limitation.
Open-loop policy evaluation covers five RoboLab policies over six tasks (300 episodes). Generated success tracks the reference ranking. A real-world cloth-folding replay is a qualitative check, not a full closed-loop study.
A Human’s Take
Conditioning a video model on “where the metal actually goes in the picture” is the kind of interface I can inspect. The Cosmos ablation is the useful bit: swap 6D actions for pixel tracks on the same backbone and the error drops.
RoboLab correlation of 0.96 is open-loop replay, not a robot choosing its next move from a dream. Fine. If you want a world model that transfers across arms and hands, start by refusing to hide the kinematics in a joint vector the pixels never see.