GERT: Graph Encoder Representations from Transformers

Knowledge Graph (KG) embedding has long been the dominant approach for reasoning over incomplete KG. However, they are primarily tailored for addressing simple link prediction queries. Constrained by their shallow and static architectures, they struggle to handle the more complex queries prevalent across various domains, such as those involving logical connectives like conjunction, disjunction, and existential quantifiers, as well as complex logical queries featuring unknown intermediate entities. In this study, we introduce the Context-aware Dynamic Graph Encoder (GERT). We propose a subgraph extraction technique that enables GERT to extract contextually relevant subgraphs from the knowledge graph using corpora, thereby providing a foundation for predicting query answers for complex queries. Furthermore, we introduce a semantic positional embedding for entities within the subgraphs, based on corpora, to denote the relationships of entities within the subgraphs in the corpora. Extensive experiments conducted on two benchmarks demonstrate that GERT outperforms baseline methods relying on static embeddings and advanced encoders across nine reasoning tasks.