Spatiotemporal attention-driven joint reconstruction of elevator audio and vibration signals via dual-modality fusion

The elevator system will generate complex voice and vibration signals during operation, and these signals contain rich operating status information, so the reconstruction of dual-modal signals is of great significance for fault diagnosis and condition monitoring. The poor synchronization and large semantic heterogeneity of elevator voice and vibration signals greatly hinder the performance of fault monitoring systems. In order to solve the difficulties in the reconstruction of bimodal signals such as poor temporal synchronization and large differences in cross-modal structures, this paper proposes a novel two-branch reconstruction model called STAViR-Net (Spatiotemporal Attention-based Vibration–Audio Reconstruction Network). The model expands on the basis of the existing spatiotemporal attention mechanism, and designs a modal-specific encoder and a shared cross-modal attention fusion core. Specifically, STAViR-Net uses a multi-scale temporal convolutional encoder (TCE) to extract the long-range dependent features in audio and vibration signals, and then aligns the two modal features in the time and frequency dimensions through a hierarchical spatiotemporal attention module (STAM) to achieve deep fusion. At the same time, the Cross-Modality Consistency Loss function is introduced to enhance the collaborative expression ability of modality while retaining the unique structural features of modal, so as to significantly improve the reconstruction quality and stability. Experimental results show that STAViR-Net improves the signal reconstruction accuracy by about 17.6% compared with the current mainstream single-modal and multi-modal reconstruction methods.