Research on coal gangue identification based on multimodal fusion and multidomain collaborative simulation

To ensure accurate identification and control of coal and gangue during top-coal caving mining, this study proposes a multimodal information fusion method integrating vibration data, infrared images, and RGB images. The vibration signals were transformed into time–frequency spectrograms using the Continuous Wavelet Transform (CWT), and a Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP) was employed for data augmentation to mitigate sample scarcity. Comparative experiments between early and late fusion strategies revealed that the late fusion approach based on the ResNet architecture yielded superior performance. With the optimal combination of vibration spectrograms (ResNet-18), infrared images (ResNet-50), and RGB images (ResNet-18), the model achieved a favorable balance between high accuracy and computational efficiency. Finally, a multi-domain co-simulation control system was developed for verification, demonstrating an average response time below 0.66 s under various rock-mixing ratio conditions. The proposed framework offers an effective technical solution for high-efficiency, clean coal production.