Dual-Path LSTM and Cross-Attention Network for Richer Image Captioning

In recent years, image captioning through deep learning is becoming of great need for multiple applications such as aiding the visually the impaired, enhancing data retrieval. Recently, it is being used with Large Language Models without requiring intensive resources. The development of modern approaches and layers lead to the attention mechanism that is being heavily utilized in various modern deep learning architectures but this does not fully replace the Long Short-Term Memory (LSTM) models just yet. The work in this paper proposes a dual-path captioning architecture that leverages both LSTM-based sequence modeling and Multi-Head Attention-based cross-modal fusion by introducing a two-stage cross-attention model — The principal of the proposed model is that the text in the image is first grounded, then refining that grounding via the LSTM’s hidden states. So, richer, context-aware caption generation is enabled. The proposed model outperforms single-path models on the Flickr-8K dataset on the BLEU 1 score by scoring 0.73.