Reliable Cross-Condition Fault Diagnosis for Structural Systems via Pretrained Multimodal Fusion of Vibration and Infrared Data

Abstract Reliable fault diagnosis of load-bearing structural components in engineering systems remains challenging under varying operational and environmental conditions, where data reliability and modality consistency often degrade. This paper proposes a cross-condition fault diagnosis framework for structural systems, leveraging a pretrained multimodal fusion of vibration and infrared measurements. First, dual-channel vibration signals—reflecting structural dynamic responses—are transformed into three-channel pseudocolor time–frequency spectrograms using short-time Fourier transform and then fused with raw infrared thermal images. A shared contrastive language-image pretraining visual encoder extracts high-dimensional semantic embeddings from both modalities. To enhance physical interpretability, the framework incorporates the root-mean-square of vibration as a physically meaningful scalar feature, constructing a physical-semantic dual-path representation space. To address the challenge of modal reliability shift under varying loads—a critical source of uncertainty in structural health monitoring—a standardized block-weighted fusion strategy is introduced. This strategy employs explicit coefficients to adaptively adjust the contribution of each modality and establishes a robust decision boundary via support vector machines. Experimental validation on two-stage helical gearbox test rig, as a representative rotating structural subsystem, demonstrates an average diagnostic accuracy of 94.40%. The results show that the proposed method integrates structural dynamic features (from vibration) with thermodynamic features (from infrared), providing a more comprehensive and interpretable physical description of fault evolution in structural components. This study offers effective technical support for risk-informed predictive maintenance and condition-based reliability assessment of civil and mechanical infrastructure under uncertain operating environments.