A Transformer-Based Model for Image Caption Generation with Memory Enhancement

This study proposes a convolution-free transformer-based method for generating accurate descriptions of images. A Vision Transformer is utilized as the primary encoder, replacing traditional CNN, and a Meshed Memory Transformer is integrated for generating captions. The Vision Transformer splits images into fixed-size patches for extracting visual features, which are further processed by a memory-augmented encoder to integrate prior knowledge. It ensures key features are preserved and reused and the meshed decoder combines the text embedding with the outputs from each encoding layer to generate more contextually rich and accurate captions compared to conventional approaches. The model's performance is evaluated on the Flickr30K and the MSCOCO datasets, demonstrating that the Meshed Memory architecture in our approach improves both the image encoding and caption generation step. Our model outperforms the conventional “CNN-LSTM” models and “CNN-Transformer” methods, achieving a 1.02% improvement in METEOR and 1.31% in SPICE on the MSCOCO dataset. Additionally, on the Flickr30k dataset, our approach exhibits superior performance across most metrics, notably a 6.37% increase in the METEOR score.