A Style-Pooling Vision Transformer for Underwater Object Classification Based on Audio-Visual Image Fusion

To enhance underwater object classification performance, we propose an audio-visual image fusion method using Vision Transformers (ViTs) equipped with style pooling-based recalibration modules (SPRMs). Our network termed as Fusion-ViT consists of two branches: one for audio image input and the other for visual image input. Each ViT branch incorporates a SPRM to improve feature representation. After feature extraction from both branches, the outputs are fused using channel-wise addition and then fed into a linear classifier for classification. We evaluate FusionViT on a custom underwater object dataset comprising paired audio and visual images. Extensive comparative and ablation experiments demonstrate the effectiveness of our audio-visual image fusion classification method. Notably, FusionViT achieves superior accuracy compared to methods that do not incorporate audio-visual fusion, all while maintaining a comparable model size. This underscores the advantages of fusing heterogeneous features for improved image classification. To the best of our knowledge, this is the first ViT architecture to achieve this.