Unified Contextualized Knowledge Embedding Method for Static and Temporal Knowledge Graph

Recent years, there is a growing interest in knowledge graph embedding (KGE), which maps symbolic entities and relations into low-dimensional vector space to effectively represent structured data from the knowledge graph. In addition, the concept of temporal knowledge graph is proposed to document dynamically changing facts in the real world. Existing works attempt to incorporate temporal information into static KGE methods to accomplish temporal knowledge representations. However, existing static or temporal KGE approaches focus on the single query fact and ignore the query-relevant contextual information in the graph structure. This paper moves beyond the traditional way of scoring facts in distinct vector space and proposes a unified framework with pre-trained language models (PLM) to learn dynamic <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">co</b>ntextualized <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</b>tatic/<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</b>emporal <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</b>nowledge <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">g</b>raph <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e</b>mbeddings, called CoS/TKGE. Given the query-specific subgraph, our model transforms it into an input sequence and uses the PLM to obtain the contextualized knowledge representations, which is flexible adaptive to the input graph contexts. We reformulate the link prediction task as a mask prediction problem to fine-tune the pre-trained language model. And the contrastive learning technique is employed to align dynamic contextual embeddings with static global embeddings. Experimental results on three widely used static and temporal KG datasets show the superiority of our model.