Segmentation Similarity Enhanced Semantic Related Entity Fusion for Multi-modal Knowledge Graph Completion

Multi-modal Knowledge Graph Completion (MKGC) aims at leveraging multi-modal information to infer missing objective facts in incomplete multi-modal knowledge graphs, thereby significantly enhancing their expressive capabilities. The segmentation of semantic data, including image segmentation and word-level descriptions, often contain implicit relationships between entities that are frequently overlooked by existing methodologies, thus limiting the effectiveness of reasoning tasks. Therefore, we propose a novel completion inference method based on fine-grained semantic segmentation, which enhances reasoning capability by utilizing implicit relationships between entities. Primarily, we introduce the concept of Semantic Related Entity (SRE) and a novel SRE selection algorithm, which captures the semantic neighboring relationships of entities based on segmentation semantic similarity to fully exploit the semantic association information. Subsequently, we propose a Multi-modal Related Entity Fusion Transformer (M-REFT) model to effectively utilize SREs from semantic modalities and neighbors from structural modality for completion inference. The M-REFT employs a hierarchical Transformer architecture to encode the fusion modality representation between each entity and its SREs, and then decode the triplet representation with the neighbor information to identify missing entities in incomplete triplets. We conducted extensive comparative experiments with several state-of-the-art models on three datasets, demonstrating the significant performance advantages of M-REFT. A series of ablation experiments and case studies further validate the rationality and necessity of the SRE concept and the SRE selection algorithm.