Robotics

Near-Sensor Tactile Depth in 0.211 ms Beats the Host PC

Robb Harlan 5 min read

Visuotactile fingers can see contact geometry, but most of them ship raw video to a host, run a Poisson solve in software, and wait on USB and the OS scheduler. Near-sensor Computing for Rapid Visuotactile Perception, posted August 6, 2026 (arXiv:2608.05725) from ShanghaiTech, moves the whole spectral Poisson reconstruction onto the sensor node.

Near-sensor visuotactile architecture compared to biological reflex and host GPU path
Biological reflex analogy vs host path vs on-chip Poisson reconstructor. Source: arXiv:2608.05725 HTML.

The hardware claim

A GelSight-style sensor (up to 400 fps) streams into a Xilinx Zynq UltraScale+ FPGA. Photometric stereo turns RGB into gradients; a fully streaming spectral Poisson core (DST-based, no iterative convergence) emits depth. At 166 MHz, the first depth sample of each 128Γ—128 frame appears after 35,107 cycles β€” a fixed 0.211 ms latency, same every frame over 1,000 timed frames.

Core logic power is estimated at 347 mW (FPGA PL simulation); a 45 nm ASIC estimate is 324 mW including SRAM. Across 15 contact geometries, hardware depth differs from a double-precision reference by 0.17% of peak contact depth after scale/offset alignment. Theoretical throughput at 166 MHz is about 10,133 frames/s for 128Γ—128; measured DMA streaming hit 10,106 fps of correct output.

Reflex that closes at the finger

A threshold on streaming depth asserts a GPIO that triggers the same servo stack used by a GelSight Mini + host baseline. High-speed camera timing: contact-to-motion 28.3 Β± 4.9 ms near-sensor (n=50) vs 169.9 Β± 27.8 ms host path (n=81) β€” about 6Γ— faster. Contact-to-command alone is 10.2 Β± 3.2 ms vs 148.6 Β± 28.3 ms. Servo mechanics eat most of the remaining near-sensor time (~18 ms). The paper notes human nociceptive withdrawal reports of 65–137 ms as a biological reference band.

Latency distributions and power comparison across FPGA, CPU, GPU, Jetson
Fixed FPGA latency vs long-tailed host/GPU/Jetson distributions; power envelopes. Source: arXiv:2608.05725 HTML.

A Human’s Take

This is the kind of systems paper I trust: fixed cycle counts, same actuator on both paths, camera-timed contact. If you care about slip reaction or contact-rich hands, deterministic sub-millisecond geometry on-chip is more useful than another soft-skin render. Scale limits above 256Γ—256 are still memory-bound β€” but the reflex numbers already make the host USB path look slow.

Sources

Tags: Robotics