Enhancing Image Understanding in Automatic Captioning using Spatially-Aware Transformer Architectures

Automatic captioning of images (ACI) is a sophisticated methodology combining image analysis and text generation, with the attention mechanism playing a critical role in identifying key image elements for elaboration. While transformer-based architectures have proven effective in text analysis and translation, their application to image captioning has been challenged by the structural disparity between image semantics typically identified by object detection models and sentence words. To bridge this gap, we introduce the Image Transformer, a novel model featuring a reformed encoding transformer tailored for spatial relationships among image regions and an implicit decoding transformer. This adaptation significantly enhances the standard transformer architecture, making it more suitable for image structures. Our model sets new state-of-the-art performance benchmarks on both online and offline MS COCO dataset testing platforms by utilizing regional features as inputs, representing a substantial advancement in ACI. Experimental results show that our spatially-aware transformer architecture achieved a BLEU-4 score of 38.4, a CIDEr score of 128.4, and a METEOR score of 27 on the MS COCO dataset, outperforming baseline methods significantly. Additionally, the model demonstrated robust performance with a 4.2% accuracy increase on the ImageNet dataset, validating its effectiveness across diverse scenarios. Its robust performance across diverse scenarios demonstrates its potential for broad application and substantial advancements in automatic image captioning.