Graph-based deep fusion for architectural text representation

Amidst the swift global urbanization and rapid evolution of the architecture industry, there is a growing demand for the automated processing of architectural textual information. This demand arises from the abundance of specialized vocabulary in architectural texts, posing a challenge for accurate representation using traditional models. To address this, we propose a novel fusion method that integrates Transformer-based models with graph neural networks (GNNs) for architectural text representation. While independently utilizing Bidirectional Encoder Representations from Transformers (BERT) and the robustly optimized BERT approach (RoBERTa) to generate initial document representations, we also employ term frequency-inverse document frequency (TF-IDF) to extract keywords from each document and construct a corresponding keyword set. Subsequently, a graph is created based on the keyword vocabulary and document embeddings, which is then fed into the graph attention network (GAT). The final document embedding is generated by GAT, and the text embedding is crafted by the attention module and neural network structure of the GAT. Experimental results from comparison studies show that the proposed model outperforms all baselines. Additionally, ablation studies demonstrate the effectiveness of each module, further reinforcing the robustness and superiority of our approach.