HuRo Paints a Robot Over 630,000 Human Videos
Human video is cheap. Robot video is not. HuRo tries to spend the cheap pile as if it were robot data.
A team at Yonsei University and RLWRLD (arXiv:2609.10706, submitted September 9, 2026) builds a pipeline that strips the person out of egocentric clips, overlays a rendered ALLEX humanoid, and retargets hand motion into robot actions. The resulting HuRo set is about 630,000 robotized episodes and 142 million frames from five sources: EgoDex, EgoVerse, Ego4D, Ego10K, and EPIC-Kitchens. That is about 1,317 hours at 30 fps.
Overlay plus actions, not just pretty fakes
Annotation fills in camera geometry and 3D hands, chunks manipulation, and captions with a VLM. Action conversion uses PyRoKi inverse kinematics. Visual conversion uses SAM2 and ProPainter to erase arms, then Isaac Sim to composite ALLEX.
ALLEX is bimanual: two 7-DoF arms, two 15-DoF hands, 2-DoF neck, 2-DoF waist. The policy is GR00T-N1.6-3B with a from-scratch action head, H=40 action chunks, 80k pretraining steps at batch 2048.
The claim is joint observation-and-action robotization at pretraining scale, not task-matched demo conversion.
Scale shows up on the table
After finetuning on small real-robot sets, overall completion on four ALLEX tasks moves from 51.5% with no HuRo pretraining to 80.3% at 100%. In-distribution: 68.1% → 88.4%. Out-of-distribution (spatial and visual shifts): 34.9% → 72.2%.
Tasks: apple pick-and-place (43 demos), cup stacking (40), cup-noodle handover (16), microwave loading (20).
A no-overlay variant that keeps human pixels but the same retargeted actions matches ID (89.4% vs 88.4%) and loses OOD (55.7% vs 72.2%). A 10% overlay subset beats full no-overlay on OOD (59.5% vs 55.7%).
End-to-end pretraining with retargeted actions beats transferring only the visual pathway. On a separate diverse pick-and-place, visual-plus-action hits 61.1% ID / 50.0% OOD. Visual-only is modest over no pretraining.
Limitations are not small. Overlay does not model occlusion against scene geometry. A five-source audit found only 55.2% of trajectories free of detected non-grasp self-contact. These clips are pretraining supervision, not executable demos. The pipeline takes about 8–10× clip duration on one RTX 5090.
A Human’s Take
Painting a robot onto Ego4D is the kind of hack that either becomes the default data engine or looks silly in a year. The OOD gap between overlay and no-overlay is the part I trust. If the policy has never seen a white arm in the frame, it should not be shocked by one. I still want a collision-aware retargeter before anyone calls 630,000 episodes “robot data.”