Modal Invariant Joint Representation Learning for Missing Multimodal Ship Image Classification

Multimodal image classification and recognition of ships can help to develop the field of intelligent traffic on ships.However, a portion of modalities is sometimes missing in specific ship identification, which can lead to the failure of traditional multimodal approaches. In order to address this problem, the current state-of-the-art approach for missing modality correlation is to directly map the available modality-specific features into a space with the aim of extracting shared potential representations that are robust to all modal combinations.In this paper, a Modality-Invariant Joint Representation Learning (MIJR) is proposed, where robust features are first extracted by Modality-Invariant Feature Learning (MIFL), and then the modal information available in the missing case is fully explored by the Transformer-related Feature Extraction Module and Modal Weight Fusion Module. This enables the performance to be maintained even when a portion of the modal data is missing. Our proposed method achieves an accuracy of 88.13% even when 30% of the public VAIS dataset is missing.