CosmoH2G Turns Flips and Rotations Into Gripper Motions
Most “copy this human video” pipelines choke when the object rotates or flips. CosmoH2G, posted 7 September on arXiv by Hongxiang Zhao, Mutian Xu, and colleagues at CUHKSZ, FNii-Shenzhen, and GenuX, is a paired hand-to-gripper dataset built for those motions.
6,189 episodes. 1,254 objects. Handheld UMI grippers mimicking the same contact patch, orientation, and path as the human hand.
Why a new dataset
Rule-based retargeting (thumb–index midpoint as the grasp) fails on natural grasps. Object-flow methods lose tracked points when a face rotates out of view. The authors’ protocol forces in-plane rotations and vertical flips, varies grasp type and contact area, and keeps start/end object poses aligned between hand and UMI takes. Pairs below 0.9 trajectory similarity are dropped; retained pairs average 0.957.
Object mix from their Table 1: toys 33.33%, decor 27.88%, food 23.03%, daily necessities 12.12%, beauty 3.64%. Code and dataset are listed as coming soon on the project page.
Two stages, then a cleanup
End-to-end pose sequences drifted. Their fix:
- Stage I predicts only start and terminal 6-DoF gripper poses from hand and object points.
- Stage II learns orientation from the paired data. Translation is initialized from the hand path (aligned contact regions) and then optimized for contact, smoothness, and inverse-kinematics feasibility.
Test split: 186 cases on 40 unseen objects. Simulation is GalaxeaManipSim on an R1 Lite. Hardware is the same wheeled dual-arm.
| Method | Sim success | Real success |
|---|---|---|
| MimicFunc | 56.45% | 43.01% |
| 3DFlowAction | 44.62% | 36.02% |
| Im2Flow2Act | 57.53% | 51.61% |
| Track2Act (retrained) | 77.42% | 60.22% |
| CosmoH2G | 83.87% | 70.43% |
Real-robot drops: objects slip during large rotations, or get dragged before they leave the table.
Limits they list: open-loop, no live collision avoidance, pick-and-place only. A one-stage model is left for when the paired set is larger.
A Human’s Take
The useful part is the collection protocol, not the transformer. If you force flips in the data, the mapping has something to learn. Seventy percent on a real R1 Lite with unseen objects is enough to keep watching. Closed-loop is the next receipt I want.