Post Hoc Multi-Granularity Explanation for Multimodal Knowledge Graph Link Prediction

The multimodal knowledge graph link prediction model integrates entity features from multiple modalities, such as text and images, and uses these fused features to infer potential entity links in the knowledge graph. This process is highly dependent on the fitting and generalization capabilities of deep learning models, enabling the models to accurately capture complex semantic and relational patterns. However, it is this deep reliance on the fitting and generalization capabilities of deep learning models that leads to the black-box nature of the decision-making mechanisms and prediction bases within the multimodal knowledge graph link prediction models, which are difficult to understand intuitively. This black-box nature not only restricts the promotion and popularization of multimodal knowledge graph link prediction technology in practical applications but also hinders our understanding and exploration of the internal working mechanism of the model. Therefore, the purpose of this paper is to deeply explore the explainability problem of multimodal knowledge graph link prediction models and propose a multimodal post hoc model-independent multi-granularity explanation method (MMExplainer) for multimodal link prediction tasks. We learn the importance of each modality through modal separation, use textual semantics to guide a heuristic search to filter candidate explanation triples, and use textual masks to obtain explanation phrases that play an important role in prediction. Experimental results show that MMExplainer can provide coarse-grained explanations at the modal level and fine-grained explanations in structural and textual modalities, and the relevance index of the explanations in model decision-making is better than that of the baseline model.