Combining Region-Guided Attention and Attribute Prediction for Thangka Image Captioning Method

To address the challenges of insufficient artistic style feature extraction in Thangka image captioning and confusion in understanding key details of character features during decoding, we propose a Region-Guided Feature Enhancement and Attribute Prediction (RGFEAP) method for Thangka image captioning. The image feature enhancement encoder, based on region guidance, utilizes a region-guided module with a distance-weighted strategy to enhance the feature representation of the central sacred elements in Thangka images.AThangka feature enhancement encoder is designed to further refine the regional feature vectors, which are then fused with the global features extracted by CLIP through multi-scale convolutional fusion, incorporating richer object-related information for Thangka images.Furthermore, to enhance the detailed representation capability in generating long-sequence captions for Thangka images, we design an attribute predictor. This predictor introduces more detailed information into the model by utilizing feature maps from four different convolutional blocks within the region-guided module. Experimental results of RGFEAP on the Thangka dataset demonstrate significant improvements compared to the baseline model ClipCap, with increases of 14.0%, 17.7%, 185.7%, and 11.5% in BLEU-1, BLEU-4, CIDEr, and METEOR metrics, respectively. On the COCO dataset in the natural domain, RGFEAP achieves comparable performance to other state-of-the-art models, showcasing strong adaptability.