Multi-Modal Knowledge Graph Representation Based on Counterfactual Data Enhanced Learning Link Prediction*

For most true multi-modal knowledge graphs, the missing links in the learning graph are very important. Most of the existing methods are based on the observed structure, text, and image information to determine whether a link exists. However, these methods can only learn to show correlations between observational data and the presence of links, and cannot capture potential causal associations between graph structure and the presence of links, text information and the presence of links, and image information and the presence of links. In this paper, we explore corresponding causality through several counterfactual questions: “If the structure of the entity, the text, the image, and the observation are different, does the link exist?”. The multi-modal knowledge graph is enhanced by constructing counterfactual-links to answer counterfactual questions. In order to generate counterfactual links, a causal model is designed to observe the existence of data and links, and a neighborhood division method in multi-modal maps is designed. The model takes the multi-modal information of the given entity pair as the context, the graph structure information, text information and image information as the processing, and the existence of triples as the result. We propose to use observation data and counterfactual triples to represent nodes on multi-modal knowledge graph. Experiments on benchmark tests show that our approach is effective.