Multi-Modal Knowledge Graph Completion Enhanced by Muti-Level Attention Mechanisms

Multimodel knowledge graph completion (MKGC) aims to predict missing entities in MKGs. Differing from prior Single-model knowledge graph completion efforts, MKGC leverages rich textual descriptions, images, and other multi-model information associated with entities. Traditional MKGC methods often use simple concatennation, weighted fusion, or tensor-based approaches for multi-modal information fusion, but they fail to capture the in-tra-modality and inter-modality dynamics. Moreover, traditional models like TransE and TransH, based on translational operations, struggle with complex relationships such as 1-N, N-N. To address these issue, this paper proposes a multi-modal information fusion enhanced by multi-level attention mechanism and relation-specific graph attention network method for MKGC. Firstly, obtaining text, attribute, and image embeddings of entities by leveraging pre-trained modals such as BERT and Vision Transformer. Secondly, using low-rank tensors for multimodal fusion and introducing self-attention and cross-attention blocks to capture complementary information within and across modalities respectively. Finally, constructing a multi-source node graph and a relation graph attention network is employed to capture the structural information of the graph. The ConvE modal is used as a decoder, and the performance is evaluated through link prediction tasks on real datasets. Experimental results show that new method out-performs other baselines, such as TransE, R-GCN, RSME, KG-BERT.