A RAG-enhanced multi-agent LLM/VLM framework for auditable and region-aware structural health monitoring of a historic masonry structure

Abstract Historic masonry structures require monitoring workflows that are non-destructive, interpretable, and robust to field variability. This paper presents a proof-of-concept, RAG-enhanced structural health monitoring (SHM) workflow demonstrated on a single historic masonry minaret case study, in which a manager–orchestrator coordinates signal analysis, visual analysis, and cross-modal fusion. Vibration data from four sensors installed on this minaret are processed through a reproducible pipeline that includes data cleaning, PSD-based baseline characterisation, adaptive windowing, robust normalisation, and anomaly screening with a 1D convolutional autoencoder trained on healthy reference data (Measurements 1–12), followed by evaluation on Measurements 13–20. Localisation is strengthened through relation-based autoencoders on sensor pairs and sensor-to-zone damage mapping. In parallel, a VLM analyses sensor-linked inspection images from the same case study, while a curated retrieval-augmented structural memory package for the monitored minaret (geometry, materials, baseline dynamics, FE notes, and inspection playbook) supports context-aware interpretation. A three-agent decision layer (Agent A: LLM+RAG, Agent B: VLM+RAG, Agent C: fusion + RAG) reconciles agreements and conflicts between signal- and image-derived evidence to produce case-specific global and zone-level risk assessments. The study compares with-RAG and without-RAG analysis modes within this single deployment and reports structured, traceable outputs, including JSON artefacts, per-image findings, per-sensor summaries, narrative reports, and final risk maps. Its main contribution is to demonstrate how signal-derived anomalies, image-derived evidence, and retrieval-grounded structural memory can be combined within a traceable multi-agent SHM workflow for one monitored historic masonry minaret.