AdvDex Maps Human Hand Demos Onto Robot Fingers
A Monday arXiv paper from Zhejiang University, Shanghai Innovation Institute, Fudan, SJTU, and Paxini Tech tries to stop treating every robot hand as its own closed dataset. AdvDex is a vision-language-action setup that trains on human and robot demonstrations in one shared action language.
The authors’ first move is data. They introduce OmniShare, which the abstract describes as more than 100k trajectories across 500 tasks and 700 objects, with kinematic supervision and tactile measurements meant to cut the need for robot teleoperation.
One wrist pose, 15 finger slots
The second piece is the Joint-Aligned Action Space (JAAS). It is a canonical command made of an SE(3) wrist pose plus 15 finger joints, three 3-DoF Euler joints per finger.
That mapping is functional, not anatomical. The paper assigns:
- 51-DoF MANO human hand states into the same 15 slots
- 19-DoF dexterous robot hands into corresponding slots
- a 7-DoF arm plus parallel gripper, with the 1-DoF jaw written into two finger slots
Unused slots get masked in the loss. The idea is one action expert, many bodies.
Hide the hardware in the pixels
Shared visual encoders can cheat by recognizing “this is a glove” instead of “this is a cup.” AdvDex adds a domain discriminator on the vision-language token, hooked through a Gradient Reversal Layer, so the encoder is punished for leaking embodiment identity.
The action expert is a Diffusion Transformer. Training is two-stage: pre-train on a mix of OmniShare, VITRA-1M internet videos, and a subset of Open X-Embodiment, then post-train on robot data.
The physical testbed is a Paxini Tora with DexH13 19-DoF hands. The authors collected 1,000 robot teleop trajectories on five tasks (single-object grasp, multi-object grasp, pour, push cube, stack bottle) and scored 20 trials per task. The paper says the full model matches or beats π0.5 and VITRA on those seen tasks, that training from scratch hurts, and that dropping OmniShare or the adversarial branch hurts more on unseen objects.
A co-training test keeps evaluation tasks only in the human set. The policy has to transfer doll-in-box, button press, bottle move, and tool use from human footage. The teaser figure on the paper reports 67.9% average real-world success for AdvDex versus 59.3% and 58.6% for the two baselines, and 51.3% on skill-transfer tasks versus 27.5% and 32.5%. Those bars live on the paper figure, not an outside lab.
A Human’s Take
I like a shared 15-joint vocabulary more than another “we teleoped our own hand.” If JAAS really lets a gripper demo teach a DexH13, that is a useful plumbing trick. The teaser bars are still one lab, one Paxini stack, and a lot of human-glove footage. I’ll believe the zero-shot transfer when someone besides the authors runs the same split on a Shadow or an XHand.