Bi-CANet: Bidirectional Contextual Alignment with Multimodal Decoder for Image Captioning

Image captioning is an approach for precisely summarizing image contents using natural language processing. Leveraging cutting-edge feature extraction, multimodal encoding, and fusion strategies, visual and textual fragments are correlated within a latent space. These multimodal segments are analyzed by a deep sequential model for phrase decoding. The widely used CNN-LSTM encoder-decoder pipeline is central to image captioning frameworks. However, current architectures suffer from two major limitations: (1) existing frameworks employ a unidirectional fusion and sequential decoding process, failing to capture the semantic context against both the preceding and following tokens; and (2) static multimodal translation fails to grasp and align critical information accurately, limiting its effectiveness when training on diverse datasets with varied visual and linguistic samples. Consequently, the models fail to capture the expansive and intricate context, leading to the production of short and abstract captions. To address these limitations, we propose Bi-CANet, an end-to-end bidirectional multimodal contextual alignment network for image captioning. Bi-CANet employs a dual-directional fusion of visual details with textual phrases along with a bidirectional decoder, thereby aligning the rich semantic and spatial context over both the preceding and subsequent phrases. Experimental analysis shows that Bi-CANet is more effective in generating comprehensive captions when trained on diverse datasets. Specifically, it achieves BLEU-1 scores of 54.02%, and METEOR scores of 33.11% on the Flickr 8K dataset.