Zero-Shot VLM Controllers Often Aren't Looking at the Camera
If a vision-language model can “drive” a sim by always hitting the brake, it will look smart on the leaderboard. That is not the same as seeing the road.
arXiv:2608.06154 (submitted Aug 6, 2026) stress-tests zero-shot vision-language control with an input-ablation battery: blank frames, noise, shuffled frames, text-only prompts, lane-axis mirrors, and non-visual baselines. Across nine direct-action models, six structured local VLMs, and an exploratory VLM–MPC stack, the authors score 32,874 calls over two embodiments and three simulators.
The ugly baseline
On highway-env, a constant-slow policy beats a scripted geometric controller on distance, reward, and crash rate while consuming no image. Several edge models collapse to near-constant actions. Hosted models split into regimes: some grade lead gap by distance (e.g. Qwen2.5-VL-72B with mutual information 0.408 bits on the gap), some only notice that an image token exists, and some disagree with themselves more often than they disagree with a blank frame.
Lateral grounding is the hard miss. Models that sort longitudinal hazard still fail to swap LEFT/RIGHT under lane-axis reflection. None of the six local VLMs pass the joint longitudinal + lateral gate. An image-only deterministic positive control recovers lead gap at 0.090 m MAE with exact mirror equivariance — so the pixels carry the information; the modular VLM stack is what fails.
What still works
A leakage-controlled symmetry-consensus guardian freezes Gemma 4-12B and Qwen3.5-9B from 16 calibration frames and takes a 2-of-4 hazard vote over original and mirrored views. On 272 held-out frames it reaches 0.954 balanced accuracy (episode-cluster bootstrap 95% CI [0.895, 0.990]). Abstaining on ties lifts committed balanced accuracy to 0.973 at 0.824 coverage. Offline modular replay that keeps deterministic lateral authority hits 0.934 action agreement and exact mirror equivariance.
An MPC hierarchy does not fix bad intent: goals fall from 23/25 (MPC-only / oracle) to 8/25 when the VLM supplies intent, even when low-level tracking is competent.
Code and artifacts: https://github.com/drdecurto/VLControl.
A Human’s Take
This paper is the cold shower the “just put a VLM in the loop” demos needed. Aggregate reward is a terrible proxy for perception when the simulator pays you for freezing. I care about the constructive half: consensus + mirror checks + abstention as a longitudinal hazard module, not as the whole controller. If your stack can’t pass a blank-image and mirror test, you do not have vision-based control — you have a slow prior with a camera for branding.