An intelligent inspection method for power equipment based on multimodal data fusion

To address the incomplete information issue of single-modal data in power equipment inspection, this study proposes an intelligent inspection method based on multimodal data fusion. By integrating text, infrared images, and audio—three types of heterogeneous data—a cross-modal feature fusion and condition diagnosis model is constructed. In the feature extraction stage, hierarchical bidirectional long short-term memory (BiLSTM) is adopted for text semantic features, an improved ResNet-50 network for multi-scale thermal features in infrared images, and time-frequency analysis combined with time-domain features for audio fault detection. For fusion, an attention-based dynamic feature weighting method is proposed, enabling cross-modal correlation modeling through semantic alignment and a gating network. Experimental results demonstrate that the multimodal fusion model achieves a fault diagnosis accuracy of 64.3%, surpassing single-modal text, image, and audio methods by 25.2%, 58.1%, and 68.4%, respectively. This approach effectively overcomes the limitations of traditional inspection techniques relying on single-dimensional data.