AI

SiMDex: Mine 5% of Ego Videos, Beat Random Mixing on Dexterous Tasks

Robb Harlan 4 min read

Dumping every egocentric human clip into a VLA fine-tune is easy. Figuring out which clips help a robot hand is harder. SiMDex (arXiv:2608.04196, University of Tokyo / ByteDance Seed and collaborators, posted August 4, 2026) treats that selection as a recommendation problem.

SiMDex three-stage pipeline: recall, ranking, optical-flow re-ranking
Three-stage mine: language/pose recall, action-chunk ranking, optical-flow re-rank. Source: SiMDex project page.

How the mine works

For each robot demonstration, SiMDex runs a cascade over a pool of about 32 million egocentric human samples (derived from EgoDex):

  1. Recall β€” cheap language + hand-pose similarity
  2. Ranking β€” wrist trajectory, wrist rotation, and finger motion alignment (this subset trains the VLA)
  3. Re-ranking β€” optical-flow check so kinematic look-alikes that do not match real motion get filtered

Actions live in a morphology-agnostic space (wrist 6D pose + five fingertip positions). The authors stress that the VLA architecture and training recipe stay unchanged β€” only the human data mix changes.

The result that sells the paper

Against a strong baseline trained on the same amount of randomly sampled human data, SiMDex uses about 1.49M mined samples (<5% of the pool) and lifts overall success from 47.7% to 61.1% (+13.4 points) on three real dexterous tasks:

  • Drill β€” multi-step tool use (grasp, assemble, trigger)
  • Flick Wheel β€” fine fingertip twist/flick
  • Pick & Place β€” four object geometries

Stages are scored separately so a late failure does not hide an early skill. Project demos include side-by-side rollouts versus a GR-Dexter baseline.

Project demo reel for SiMDex task rollouts. Source: SiMDex / YouTube.

Three real-world dexterous evaluation tasks: drill, flick wheel, pick and place
Evaluation suite: tool use, fingertip dexterity, multi-object pick-and-place. Source: SiMDex project page.
Physical tools and parts used in SiMDex dexterous manipulation tasks
Physical tools and assembly parts used on the robot platform. Source: SiMDex project page.

A Human’s Take

Selection beats scale is a useful corrective in a year when every lab is hoarding ego hours. If <5% of a curated pool beats an equal-size random dump, the next bottleneck is mining quality, not disk. I want open-sourced retrieval indices and a third-party hand stack to rerun the same three tasks. Until then, SiMDex is a strong argument that β€œmore human video” is not the same as β€œbetter human video.”

Sources

Tags: AI