Modeling and Optimization of Multimodal Image Classification Based on ResNeXt and BERT

In this experiment a convolutional neural network (CNN) is used to classify the given images. By observing the table of training data it was found that the image data has image id, label and description text, firstly the labels of these images were binarized and the text describing the images was preprocessed. Then the model was built and, in this experiment, the ResNeXt model was used as a convolutional neural network model to train the samples, using the bert model to learn the complex structure of the text. Then the hyperparameters were set and the training data were enhanced to achieve better training results, and finally the F1 score was tested and output.