LiLa-WAM Trains a World-Action Model on One 24 GB GPU
Most world-action models (WAMs) either render future pixels (expensive) or build latent foresight through multi-stage pipelines (still heavy). LiLa-WAM (arXiv 2608.03701) aims at the budget nobody markets: train end-to-end on a single 24 GB GPU, keep foresight, skip language at test time.
Design in plain terms
- Frozen DINOv3 visual backbone → query adapter compresses patches into a small token set
- Foresight-Aware Action Expert: one DiT stream jointly predicts action velocity (flow matching) and a future latent supervised in DINOv3 feature space
- Visual Transition Token (VTT): offline, per task, mean of (last-frame − first-frame) global embeddings from demos — a language-free “what this task changes” vector at deployment
No pixel decoder at inference; foresight decoder is training-only. Model size reported: ~0.5B total (~0.2B trainable, ~0.3B frozen). Joint training over 50 RoboTwin tasks: about 110 GPU-hours on one RTX 5090. Inference: ~85 ms on an RTX 4090 (paper).
Benchmarks
RoboTwin 2.0 — single model, 50 tasks: average success 90.48% (paper table), competitive with multi-billion-parameter WAMs/VLAs under far less capacity.
LIBERO — four suites: average about 97.1%, matching much larger OpenVLA-style baselines in the paper’s comparison table.
Ablation (10-task subset): removing foresight loss drops 70.0% → 54.4%; swapping VTT for CLIP language costs 8.6 points.
Real Agilex Piper (four tasks, 50 evals each): full model 82.0% average vs 74.0% without foresight loss.
Code link in the abstract: github.com/teee000/LiLa-WAM.
A Human’s Take
Single-GPU trainability is a product requirement, not a brag. If foresight only works when you rent a cluster, labs will keep shipping reactive VLAs. LiLa-WAM’s VTT idea is cheeky — encode the task as a visual delta instead of English — and the ablation says foresight is doing real work. I want to see third-party replications on RoboTwin’s randomized split and whether VTT still works when demos are messy human videos, not clean teleop.