Large multimodal model assisted underground tunnel damage inspection and human-machine interaction

Artificial Intelligence is playing an increasingly important role in tunnel inspection as a core driver of the new generation of engineering. Traditional methods are difficult to directly generate human linguistic information and lack valid messages extracted from different modalities. This paper proposes Damage LMM, a multimodal damage detection model that can handle images or videos as well as text inputs, to realize fast damage identification and human-computer interaction. The visual instruction database is first created from real damage data collected using different visual sensors and captions extracted by a regional convolutional neural network. The basic language model is then fine-tuned into a specialised Damage LMM, which enhances user instructions by integrating virtual prompt injection and system messages. Finally, the enhanced prompts are processed through the tuned multimodal model to generate a detailed visual description of the damage. The performance of the method is evaluated using a real tunnel dataset, and the results show that it has better robustness and accuracy than other models in multimodal data, with an accuracy of 0.93 for the in-domain image data and a contextual correlation of 0.94. The proposed method can effectively identify tunnel defects and realize multimodal user interaction functions with a moderate number of markers and a short delay time, which will greatly help engineers to quickly obtain effective information and assess the degree of damage at the tunnel inspection site. • Innovative multimodal damage detection model that can process images/videos along with text inputs. • Specialised fine-tuning for generating detailed damage descriptions based on visual instruction database. • Achieves high accuracy and strong contextual correlation, outperforming traditional models in robustness and precision. • Enables engineers to quickly access actionable information with minimal markers and short delay times. • Realizing damage identification and enhanced human-machine interaction to improve tunnel inspection efficiency.