Fusion of Masked Attention Mechanism and Multi-Granularity Labels for Thangka Image-Text Matching

As a unique painting art, Thangka has been included in the list of intangible cultural heritage. The digital protection and research of Thangka has important social and practical significance for the development, inheritance and innovative development of national culture. This paper combines attention mechanism and multi-granularity label, and studies the Thangka image-text matching method with 30 common Thangkas of different themes as the research object. This paper proposes a novel image-text matching structure that integrates mask attention mechanism and multi-granularity label. The structure has the following improvements: 1) A named entity recognition model for Thangka text is designed to more fully mine the structure and semantic information of the text; 2) The traditional attention mechanism is improved, a mask layer is added, and the multi-label matching process is completed for the processing of unrelated data; 3) A new way of mapping images and texts. Comparative experiments and ablation experiments were conducted on the proposed method. It was found that the recall rate of the proposed method in image matching text and text matching image was improved by $3.8 \%$ and $4.9 \%$ respectively on the Thangka dataset compared with other methods. The recall rate on the Flickr30k dataset was better than all traditional VSE models in two matching tasks. The performance is not inferior to the current pre-trained models that require large-scale data training, which shows that our model works well on different data sets.