Vision‐Aided Damage Detection With Convolutional Multihead Self‐Attention Neural Network: A Novel Framework for Damage Information Extraction and Fusion

The current application of vibration‐based damage detection is constrained by the low spatial resolution of signals obtained from contact sensors and an overreliance on hand‐engineered damage indices. In this paper, we propose a novel vision‐aided framework featuring convolutional multihead self‐attention neural network (CMSNN) to deal with damage detection tasks. To meet the requirement of spatially intensive measurements, a computer vision algorithm called optical flow estimation is employed to provide informative enough mode shapes. As a downstream process, a CMSNN model is designed to autonomously learn high‐level damage representations from noisy mode shapes without any manual feature design. In contrast to the conventional approach of solely stacking convolutional layers, the model is enhanced by combining a convolutional neural network (CNN)–based multiscale information extraction module with an attention‐based information fusion module. During the training process, various scenarios are considered, including measurement noise, data missing, multiple damages, and undamaged samples. Moreover, the parameter transfer strategy is introduced to enhance the universality of the application. The performance of the proposed framework is extensively verified via datasets based on numerical simulations and two laboratory measurements. The results demonstrate that the proposed framework can provide reliable damage detection results even when the input data are corrupted by noise or incomplete.