Seeker Learns Where to Look From the Action Stream
Most “look here” modules for robot policies want extra labels: gaze, object boxes, VLM crops. Seeker, from KTH, Freiburg, and Universität Hamburg (arXiv August 13), trains the crop from the action stream alone.
A task- and state-conditioned query walks over frozen DINOv3 patch tokens, gathers evidence, and updates itself. The readout becomes a mask and a box. Downstream you can crop RGB, hide the background, or filter a point cloud. Code and a MimicGen checkpoint are on GitHub.
Attention as a search, not a keyframe
Action-derived crops usually lock onto the gripper at a motion stop. That fails when the cue is a lid across the table or a slot the fingers have not reached yet. Seeker does not pick a proxy point. Multiple readout heads are gated by the current query, so the focus can move from object to contact to spatial relation as the task proceeds.
The readout is trained with a diffusion action-prediction loss. That head is only there to supervise “where to look.” A separate policy then consumes the focused view.
The lift
Under the same RGB stack, Seeker raises average simulation success from 42.6% to 62.6%. On real robots, in-domain success goes from the best baseline’s 48.3% to 76.7%. Under lighting and background shifts it goes from 20.0% to 60.0%.
The released seeker.mimicgen.pth was trained jointly on six MimicGen tasks: coffee preparation, pick-and-place, square, stack-three, threading, and three-piece assembly. The repo also documents a guided background-swap protocol that paints new textures outside the predicted mask.
A Human’s Take
I like that nobody had to wear an eye tracker. If the action already knows where the demonstrator was looking, you should be able to recover that map. The 20-to-60 jump under lighting changes is the number I will quote: a crop that still finds the mug when the tablecloth changes is more useful than a 2-point sim bump. I want to see the same readout on a humanoid wrist cam, where the background is a kitchen, not a MimicGen table.