Deep learning-based multimodal rainfall detection using image and audio data

Accurate and timely monitoring of rainfall is essential for managing reservoir sediment, as intense rainfall accelerates surface erosion and increases sediment inflow, leading to a loss of storage capacity. However, traditional rain gauges lack spatial resolution, and radar systems are limited by terrain interference and signal noise. This study therefore proposes a deep learning-based, multimodal rainfall detection system that integrates optical images and acoustic data within an Internet of Things (IoT)-enabled framework. While image inputs capture spatial rainfall patterns, they perform poorly in low light. In contrast, acoustic sensing, based on raindrop impact sounds converted to Mel Frequency Cepstral Coefficients (MFCCs), can be used to complement classification in darkness or in settings where visibility is impaired. Both data types are processed through convolutional neural networks (CNNs) and fused at the feature level to enhance classification accuracy. The system was tested using 16,528 samples from artificial rainfall experiments and 6,445 samples from real rainfall events. The optimal model configuration, comprising a seven-layer image CNN and a three-layer audio CNN, achieved an accuracy of over 99% in controlled experiments, and 99.33% during the day and 70.56% at night in real-world conditions. Grad-CAM analyses confirmed that the model learns distinct, interpretable features and benefits from multimodal fusion. By providing high-resolution, real-time rainfall classification, the system supports improved sediment transport modelling, facilitates proactive dredging and flushing decisions, and contributes to more resilient and sustainable reservoir operations in the face of increasing climatic uncertainty.