Track4Action Distills 3D World Tracking Into Tracker-Free VLA Policies
Action labels tell a VLA what to do. Demo video also records how the 3D world changed while those actions ran. Track4Action (arXiv:2608.03727, submitted Aug 4, 2026) distills that missing geometry and motion from a frozen world-centric 3D tracker into a policy that, at deploy time, only needs the current observation.
How the distillation works
During training, Track4World encodes an aligned demonstration clip into a pooled tracker feature. Learnable track queries infer that feature from current VLA hidden states, match it in a shared space, and condition a flow-matching action head through a feature-wise gate. The tracker feature is only an alignment target — neither the clip nor the tracker is used at deployment.
That is the product-relevant claim: richer 3D supervision without shipping a 3D tracker on the robot.
Numbers from the abstract
| Benchmark | Result |
|---|---|
| Zero-shot LIBERO-Plus | 82.3% (+7.6 pts vs alignment-free variant; +3.0 vs LaMP) |
| RoboTwin 2.0 clean / randomized | 80.44% / 81.48% |
| Four physical bimanual tasks (avg) | 67.5% success (+25.0 pts vs alignment-free) |
Project page: wing0night.github.io/track4action-project-page.
A Human’s Take
I’m so here for “privileged training signal, thin deploy stack.” If you can teach geometry and visibility from video and then throw the teacher away, you get closer to policies that care how the scene moves — not just which joint command was logged. The +25-point real bimanual jump is the number that makes me want a third-party re-run.