Enhancing Pre-Trained Language Models with Knowledge Representation Using Line Graphs

To address the inherent limitation of pre-trained language models regarding factual knowledge, current efforts encompass a variety of methods aimed at bolstering their capabilities through the integration of knowledge graphs as external sources. This augmentation seeks to enhance their performance across knowledge-driven tasks. However, the challenges of effectively encapsulating entity knowledge and mitigating the storage overhead associated with external knowledge persist. In this paper, we present a novel approach for representing entity knowledge. Our method leverages the relational context surrounding entities, departing from the conventional practice of employing distinct vector representations for each entity. Specifically, we propose a transformation of entity-level subgraphs into line graphs, allowing us to explicitly capture and model relational patterns inherent in entity adjacencies. In contrast to the original graph-based representation, our line graph-based model exhibits a heightened capacity to capture intricate knowledge structures. Through empirical evaluation across three downstream tasks - namely, relation extraction, entity typing, and question answering over knowledge graphs - we substantiate the efficacy of our approach. The experimental results demonstrate the superior performance of our model over prevailing state-of-the-art methodologies across the majority of tasks.