A Case Study on Visual-Audio-Tactile Cross-Modal Retrieval

Cross-Modal Retrieval (CMR), which retrieves relevant items from one modality (e.g., audio) given a query in another modality (e.g., visual), has undergone significant advancements in recent years. This capability is crucial for robots to integrate and interpret information across diverse sensory inputs. However, the retrieval space in existing robotic CMR approaches often consists of only one modality, which limits the performance of the robot. In this paper, we propose a novel CMR model that incorporates three different modalities, i.e., visual, audio, and tactile, for enhanced multi-modal object retrieval, referred to as VAT-CMR. In this model, multi-modal representations are first fused to provide a holistic view of object features. Then, to mitigate the semantic gaps between representations of different modalities, a dominant modality is selected during the classification training phase to improve the distinctiveness of the representations and enhance the retrieval performance. To evaluate our proposed approach, we conducted a case study and the results demonstrate that our VAT-CMR model surpasses competing approaches. Further, our proposed dominant modality selection significantly enhances cross-retrieval accuracy.