AvTF: A visual-tactile fingertip with active sensing and manipulation

In the interaction between dexterous hands and the environment, perception and operation are inseparable. Perception provides information and feedback for operations, and operations in turn affect the perception. Current VTS(visual-tactile sensors) lack initiative and cannot sense the state of the operated object in the world coordinate system. The AvTF(Active visual-tactile Fingertip) introduced in this article is a vision-based fingertip (finger-shaped full-field sensing) tactile sensor that uses a camera to capture tactile images. Sensing and operating are integrated by adding active DoF (Degree of Freedom). Camera modules under different coordinate systems can provide various perspectives for sensors. Through a series of experimental verifications, AvTF has achieved real-time feedback on the grasping and manipulation of objects in the hand, providing a new solution for dexterous hands to fuse sensing and operation better.