A large language model-aided vision-based framework for automated structural displacement measurement

Accurate measurement of structural displacement is critical for effective monitoring and maintenance of civil infrastructure. In recent years, vision-based techniques have been widely adapted to structural displacement measurement due to their non-contact capabilities. However, in existing methods, selecting an appropriate region of interest (ROI) for displacement tracking remains a fundamental challenge because of the reliance on a task-specific detection model that requires extensive training data or manual ROI selection by experienced users. This dependence limits the employment of practical and adaptable vision-based structural displacement measurement systems. To address this issue, this study proposes a large language model-aided vision-based framework for structural displacement measurement that leverages the vision-language understanding capability of LLMs to identify optimum ROI. First, three state-of-the-art LLMs, guided by designed prompts, are used to identify and describe potential ROIs on the monitored structure that are suitable as targets for displacement tracking. Subsequently, an open-set object detector, Grounding DINO, is leveraged to detect potential ROIs through their descriptions. The optimum ROI is selected from all potential ROIs through a feature density analysis process. Finally, the displacement of the monitored structure is measured by inputting the optimum ROI into a Kanade-Lucas-Tomasi (KLT) tracker. The proposed method is validated through a series of experiments, demonstrating that the appropriate ROI can be automatically selected without additional model training or manual procedures, while still providing measurement results comparable to reference sensors. These results highlight the potential of integrating LLMs into vision-based structural monitoring workflow to enhance practicality and adaptability of displacement measurement framework.