Text-KG Distill: Textual Knowledge Distillation for Enhanced KG Embeddings

Knowledge graph embedding (KGE) models are widely used for knowledge graph completion by predicting the plausibility of missing triples. While these models focus on structural relationships, they often overlook valuable textual information found in literals, which can negatively impact performance. To address this limitation, we propose an approach leveraging Pre-trained Language Models (PLMs) to integrate textual knowledge into KGE learning. Our method uses knowledge distillation, where a PLM acts as a teacher to transfer semantic information to a target KGE model. By employing Sentence-BERT as the PLM, we evaluate the plausibility of textual facts and enhance the KGE model through a modified loss function incorporating Kullback-Leibler (KL) divergence. We introduce a trainable slack variable to regulate the influence of the PLM on the target model's semantic learning. Experiments on four KGE models across two benchmark datasets demonstrate that our approach effectively enhances KGE models by incorporating missing semantics from textual knowledge.