SAM-MFF: An Image Captioning Generation Method Based on Multi-layer Feature Fusion

In the task of image captioning, some models tend to primarily use deep features extracted by CNN to represent image information, but shallow features also have the potential to improve model performance. To address this issue, this paper designs an image captioning generation method based on multi-layer feature fusion. After the image is processed by the backbone network to extract features, firstly, the shallow features are input into our designed Spectral Alignment Module (SAM), which unifies the position segmentation of features on the spectral graph, making different layers consistent with deep features in feature channels while minimizing feature loss as much as possible. Then, the deep features interact with SAM-processed shallow features to generate new composite features. These new features will cross-attention with original deep features and text features respectively and mutually verify to generate more accurate image captionings. Experimental results on Microsoft COCO dataset show that the proposed method can effectively select key information at the feature dimension level to generate high-quality sentences and improve BLEU1 and CIDEr score by 0.8% and 1.7%. It can be seen that the image captioning generation method based on multi-layer feature fusion can effectively improve accuracy in describing images with text.