Language Models-enhanced Semantic Topology Representation Learning For Temporal Knowledge Graph Extrapolation

Temporal Knowledge Graph (TKG) extrapolation aims to predict future missing facts based on historical information, which has exhibited both semantics and topology of events. The mainstream methods have advanced the prediction performance by exploring the potential of topology representations of TKGs based on dedicated temporal Graph Neural Networks (GNNs). Until recently, few Language Models (LM) based methods have attempted to model the semantic representations of TKGs, however, lacking specific designs for the topology information. Therefore, we propose a Semantic TOpology REpresentation learning (STORE) framework enhanced by LMs to bridge the gap between the semantics and topology of TKGs. Firstly, we tackle the challenge of long historical facts modeling by a time-aware sampling based on semantic priors to extract concise yet precise facts. Secondly, we handle the challenge of the interaction between topology and semantics by transforming graph representations into virtual tokens that are then integrated with generated prompts and fed into LMs. Finally, multi-head attention is adopted to obtain better semantic topology representations, thereby achieving joint optimization of both temporal GNNs and LMs. Extensive experiments on five datasets show that our STORE outperforms state-of-the-art GNNs- and LM-based methods.