A vision-based tactile sensor with in-sensor computing paradigm for seamless tactile sensing and perception

Abstract Vision-based tactile sensors (VBTSs) offer high-resolution, multimodal tactile sensing capabilities, supporting precise and dexterous robotic manipulation. However, the physical separation of sensing and perception modules necessitates frequent data transfer, introducing communication overhead and potentially constraining real-time performance. In biological tactile systems, sensory transduction and preliminary signal processing occur at the periphery before information is transmitted to the central nervous system, thereby enabling efficient and timely tactile perception. Inspired by this principle, we present PixelTac, a VBTS designed under an in-sensor computing paradigm. PixelTac integrates sensing, memory, and computation within a single in-situ vision processor that combines a computational pixel matrix and a microcontroller. This architecture enables local compression of tactile data and fully on-chip processing from pixel-level sensing to force inference. Here, we show that PixelTac reproduces key features of human mechanoreception in real-time while maintaining efficient power consumption, providing a scalable and energy-efficient route to seamless tactile sensing and perception.