Heritage Building Information Management and Intelligent Querying by Multimodal Large Language Models and Knowledge Graph

Heritage buildings face challenges in documentation due to inconsistent records and complex data from historical documents, archaeological surveys, and materials. Traditionally, converting unstructured data into structured formats required significant expert effort. The advent of large language models (LLMs) has transformed heritage research by enabling the creation and maintenance of knowledge graphs. These graphs integrate diverse data sources, facilitating the preservation and study of heritage buildings. LLMs help extract and organize unstructured data, improving knowledge graph accuracy and consistency. This research proposes a comprehensive framework that integrates multimodal data, including text, images, and videos, into a unified knowledge graph. The framework employs LLMs for extracting information from textual data, the CLIP model for aligning images with corresponding text, and keyword searches for processing video content. The resulting knowledge graph is stored in a Neo4j graph database, providing an interactive platform for users to query and explore detailed information about heritage buildings. This approach not only supports academic research but also contributes to practical applications in cultural heritage conservation, enabling more efficient access to valuable information and enhancing preservation efforts. The proposed method was validated in European 'Gothic' and 'Gothic Revival' architecture by comparing the relationships between components.