Adaptive OCR-Driven Image Encoders for Enhanced Text-Based Captioning: Achieving High-Precision Descriptions in Text-Rich Visual Contexts

This work introduces a new multimodal method that improves image captioning in the TextCaps task by successfully combining visual features with text from OCR. Experimental results on the TextCaps dataset show that our approach achieves a BLEU-4 score of 25.84, a METEOR score of 23.24, a CIDEr score of 96.32, and a SPICE score of 21.10, outperforming state-of-theart methods. These advances result in more accurate and contextaware captions. Traditional methods are affected by weak blending of visual and textual information, shallow text comprehension, and difficulty in detecting object-text interaction. In order to counter these limitations, our technique suggests two attentions under a transformer model in order to support imagery modality representation. 'Master object' architecture combines the visual and textual features into the best scene graph, while visual embedding representation is enabled by Graph Convolution Network. In addition, hierarchical clustering strengthens the representation of OCR tokens in a way that OCR text and image objects share strong correlations. By improving multimodal fusion, this approach significantly improves caption accuracy and contextual coherence, advancing the state of the art of image captioning.