An Image Captioning Method Based on Transformer for Multi-feature Fusion

In image captioning, methods that generate descriptions by extracting regional visual features often overlook the rich contextual information and details contained in grid features. Additionally, Transformer models have some limitations in distinguishing between visual and non-visual vocabulary. To address these issues, this paper proposes a Transformer-based multi-feature fusion approach for image captioning. The model employs a bidirectional encoder to extract intrinsic features from both grid and regional features, and constructs a multi-level perception enhancement module to leverage semantic information from different encoder levels. Furthermore, at the feature fusion stage, this paper introduces a multi-visual feature fusion module that effectively resolves the semantic noise problem arising from multi-feature fusion. Lastly, to differentiate between visual and non-visual words, the paper also establishes a Transformer-based language model to extract semantic features. Through a dynamic attention mechanism, it adaptively captures the correlations between visual and lexical elements, guiding the generation of image descriptions. The proposed method is trained on the MS-COCO dataset and quantitatively compared with existing approaches. Experimental results demonstrate that this method can generate more accurate and richer image descriptions.