Bidirectional dense connected image caption based on transformer

In the field of image captioning, researchers have achieved significant progress by adopting an encoder–decoder architecture. Recently, most approaches rely on faster region-based convolutional neural network (R-CNN) or convolutional neural network (CNN) as the encoder to extract region-level features from input images. However, CNN has limitations in capturing semantic information, and faster R-CNN exhibits limited performance in capturing more complex contextual information and global visual relationships. To address these issues, we propose an end-to-end model based on a transformer with bidirectional dense connection feature fusion. The model first utilizes the Swin Transformer to extract grid features from the input image at four stages and then fuses these features using bidirectional dense connections based on shared attention. In the image caption generation phase, the transformer decoder with dual attention decodes the image features and introduces a multi-level supervision mechanism to improve the model’s ability to learn representations from intermediate layers, enabling the diverse generation of image captions. Through experiments on the microsoft common objects in context (MSCOCO) dataset, this model achieves excellent performance, with evaluation metrics such as BLEU-1, BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE reaching 82.9, 41.1, 30.6, 60.6, 141.2, and 24.4, respectively.