Combining Semi-Supervised Learning of Visual-Tactile Feature Fusion Shared Network for Object Recognition

In simulating human perception of the environment, intelligent systems can significantly improve decision-making performance for object recognition by integrating visual and tactile information. Current mainstream fusion strategies in this field are usually performed under supervised learning, and the tendency to simply concatenate visual-tactile features at specific layers of a deep neural network may ignore the intrinsic relationships that exist between different modalities. To address this problem, we propose a fusion shared network of visual-tactile features combined with semi-supervised learning. Aiming to process two types of input data under semi-supervised strategies and construct a common space for visual-tactile modal shared learning, which can automatically adjust the weight matrix according to the interrelationships between the visual and tactile information, which allows the model to exchange the information between the two modalities, realizing the complementary fusion of the information. In this paper, the method is evaluated on the PHAC-2 and VHAC multimodal datasets, and the results show that the method can improve the performance of visual-tactile fusion perception for object recognition by 1% to 8.7% compared to the traditional fusion strategy under supervised learning, and the performance is improved by 0.6% to 8.3% when the labeling ratio is 50% under semi-supervised learning, which demonstrates the method’s sparseness of labeling data and its effectiveness.