GEN-1.5 Learns a New Robot Task From a 3-to-12-Second Demo
Generalist’s new robot foundation model, GEN-1.5, can pick up a short manipulation skill from a single 3-to-12-second demonstration, with no gradient updates. The company posted the research on August 19. IEEE Spectrum’s Video Friday featured it this week.
The trick is what Generalist calls physical prompting: drop a sensorimotor clip into a 30-second context window, then let the model output 100 Hz action trajectories. Across 10 short-horizon tasks (jar lids, zippers, wallets, brushing a cube into a bowl), one-shot prompting from the pretrained weights averaged 59% success (±10%). Ten gradient steps on about five minutes of data per task (~50 demos) lifted that to 83% (±9%).
What “one-shot” actually is here
This is not “show it once, it never fails.” Generalist says the tasks are simple and short, and in-context skills are more brittle than fine-tuned ones. The interesting part is that they did not add an architecture for in-context learning, a meta-learning loop, or an auxiliary improvisation loss. The company says the behavior showed up after more than eight months of continuous pretraining on physical interaction data.
Other tricks in the same post:
- Compose two prompts. Unzip a pouch, then pull money. The model fills in the regrasps that were in neither clip.
- Sim prompt, real robot. A demonstration recorded only in simulation can prompt the real arm, even though pretraining contains no sim data.
- Sometimes, bare hands. A person does the task in view of the robot cameras; the robot tries it next.
The dustpan, the banana, the paper
Fine-tuned for a few minutes on brushing a block into a bowl, the model used a banana as a makeshift brush. Handed a dustpan, it scooped and dumped. After one gradient step, it pulled a piece of paper off the bowl that was never in the five-minute fine-tune set. Humanoids Daily quotes NVIDIA’s Jim Fan on why messy human data matters: keep the failed first half instead of trimming the fumble.
A single gradient step on one minute of data hit 66.5% on a held-out task. Ten steps moved weights by less than 0.15%, which is Generalist’s way of saying the model is being reminded, not rebuilt.
A Human’s Take
I’m so here for a robot you can teach with a ten-second clip instead of a week of teleop. Fifty-nine percent on short tasks is not a coworker. It is a new starting line. The dustpan scoop is the receipt I wanted: a tool that was not in the demo, used like a tool. Next I want that on a shift, not a table.