Vision-Language Fusion Network for Time Series Classification in Machinery Fault Diagnosis

In machine health monitoring, given the scarcity of real faulty data samples, traditional deep learning techniques are hindered to meet few-shot diagnosis requirements due to their insufficient generalization performance. Vision-language model (VLM) has already achieved excellent results in the fields of natural language processing and image processing. Its vast prior knowledge and powerful reasoning ability are conducive to solving the problem of machinery fault diagnosis. This paper proposes a novel fault diagnosis framework based on VLM for machine health monitoring. The proposed vision-language fusion network employs Qwen2-VL-Max as the description generator and the Qwen encoder as the text encoder. Furthermore, we transfer the original time series signals into spectrograms and further design instructions to describe the generated spectrograms via VLM reasoning. In addition, we propose a dual cross-attention residual network (DCARN) to fuse text and spectrogram features. Experiments on the HUST dataset show that the proposed model outperforms the state-of-the-art models, achieving an average overall accuracy of 95.7% and an average Macro-F1 score of 0.956. These results fully demonstrate the potential of VLM in enhancing the performance of machinery fault diagnosis.