A CNN-Transformer Approach for Image-Text Multimodal Classification with Cross-Modal Feature Fusion

This study proposes an image-text multimodal classification algorithm based on a combination of convolutional neural networks (CNN) and Transformer, aiming to solve the key problems in feature extraction and the fusion of multimodal data. In view of the differences in feature space and semantic expression between image and text modalities, respectively, adapted feature extraction modules are designed, where CNN is used to extract local features of images and Transformer is used to capture global semantic information of the text. Efficient feature extraction aligns well with the need for diverse data interpretation in wearable devices, such as recognizing gestures, monitoring vitals, and interpreting user inputs. This innovation is particularly significant in the field of the Internet of Things (IoT), where wearable devices and other connected systems rely on accurate, real-time data processing for enhanced functionality and user interaction. On this basis, a cross-modal attention mechanism is introduced to achieve deep feature interaction and alignment between modalities, thereby effectively fusing the information of the two modalities. In order to improve the efficiency and robustness of the model, a lightweight network design is adopted in the method, and data enhancement and regularization strategies are combined to improve the generalization ability of the model. This study provides a novel solution to the multimodal classification problem in theory and has the potential to expand its application in practical scenarios, especially showing good adaptability when processing complex and diverse data.