Hijacking a Sorting Robot With a Sticky Note: Physical Prompt Injection
If a robot plans with a vision-language model, the scene itself becomes a prompt. Hijacking Robots with a Piece of Paper, posted August 6, 2026 (arXiv:2608.05715), is a systematic study of physical prompt injection against VLM-controlled sorting: adversarial text placed in the camera’s view that steers the model’s reasoning stack.
The attack surface
The authors introduce a four-category taxonomy:
- Indirect signage
- Task redefinition
- Authority impersonation
- Conflict injection
They instantiate a benchmark of 20 attack prompts across three physical scene layouts and three command formulations that vary destination specificity and rule explicitness. Evaluation spans 5,670 trials on three frontier VLMs: GPT-4o, Gemini 2.5 Flash, and Qwen3-VL-32B.
Attack success rates: 27.0%, 29.4%, and 5.0% respectively. Authority-impersonating and negation attacks transferred across all three models. Reasoning-trace analysis says successful compromise is almost always conscious (99.9% acknowledgment rate) — the model sees the text and still follows it. Defenses differ by model: Gemini more often explicitly rejects; GPT-4o more often fails via perceptual inattention.
Mitigations that actually moved the needle
Three simple defenses were tested:
- Prompt-based defense — 75–100% effective, model-dependent
- Two-stage verification — 85–100%
- Pre-processing text masking — 100% in their benchmark
The authors note those defenses preserved general task capability in the benchmark, but may hurt tasks that require reading in-scene labels.
A Human’s Take
I love a paper that weaponizes sticky notes. If your warehouse humanoid trusts whatever text falls into the frame, someone will put “ignore sorting rules” on a tote. Masking is blunt but honest; the product question is how you keep label-reading jobs while closing the injection path. Ship the verification loop before you ship the smile.