UDI-3DVT: A Novel Network for 3-D Reconstruction via Uncertainty-Driven Sampling and Incremental Visual-Tactile Fusion

The fusion of visual and tactile senses allows robots to reconstruct and understand objects, aiding in downstream tasks such as object recognition and object grasping. However, most existing visual-tactile reconstruction methods typically overlook the role of visual point cloud processing in reconstruction, sampling points using generative-based methods rather than uncertainty-driven approaches. In addition, these methods only concatenate visual and tactile features, failing to fully leverage the complementary advantages of these two modalities, resulting in unsatisfactory reconstruction results. To address this issue, leveraging the open-source GelSight tactile sensor, this article proposes a novel method called UDI-3DVT, which reconstructs 3-D shapes by combining visual point clouds and multiple tactile images. Specifically, we explore the uncertainty of point clouds, performing additional sampling on high-uncertainty edge points, and propose an attention-based point cloud sampling method. Then, the framework leverages the efficiency of convolutional neural network (CNN) in image processing and the transformer’s superiority in context modeling by using CNN for tactile features and transformer for visual features. A feature fusion module based on <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${1} \times {1}$ </tex-math></inline-formula> convolution is designed to aggregate features from two branches. By gradually adding tactile features, we achieve incremental visual-tactile integration. Furthermore, since the contact-based perception of tactile sensors may cause slight deformations in objects, UDI-3DVT supports the reconstruction of both rigid and deformable objects. Extensive experiments demonstrate that our proposed framework outperforms current state-of-the-art (SOTA) methods, especially the Chamfer distance (CD) metric increased by 14.8%.