Touch Dreaming: CMU Humanoid Learns Contact Before It Happens
Carnegie Mellon, UT Arlington, and the Bosch Center for AI introduced Humanoid Transformer with Touch Dreaming (HTD)—a whole-body learning stack that treats touch as a first-class modality, not a bolt-on sensor dump. On five real contact-rich tasks, HTD posted a 90.9% relative improvement in average success rate over the stronger baseline.
The idea: dream the contact, then act
Humanoids fail contact-rich work when policies only see vision and proprioception. HTD is a multimodal encoder–decoder Transformer trained with behavioral cloning plus an auxiliary objective the team calls touch dreaming:
- Predict action chunks (as usual).
- Also predict future hand-joint forces and future tactile latents.
- Latent targets come from an EMA target encoder—no separate tactile pretraining stage.
Ablations show latent-space tactile prediction beats raw tactile prediction (~30% relative success gain). Tasks include Insert-T (tight clearance), book organization, towel folding, cat litter scooping, and bimanual tea serving.
System stack
The paper (arXiv:2604.13015) pairs HTD with:
- An RL lower-body / whole-body controller for balance under upper-body disturbance.
- VR teleoperation and human-to-humanoid motion mapping for data collection.
- Dexterous hands with distributed tactile sensing.
TechXplore quotes lead author Yaru Niu on combining full-body coordination, hand dexterity, and predictive contact understanding—the same ingredients humans use when folding cloth or inserting parts. Senior author Ding Zhao notes that vision-only policies often miss fast contact changes.
Code and materials are linked from the project site; IEEE Spectrum has featured related teleop video clips in Video Friday roundups.
A Human’s Take
Predicting how a grasp will feel a half-second ahead is the right obsession. I’m less interested in the relative-percentage banner and more in whether latent touch transfers across hands and skins. Open code helps. If touch dreaming becomes a standard BC auxiliary—the way language models got next-token prediction for free—contact-rich humanoids get a real lever.