Analysis and identification of nocturnal groaning syndrome based on multimodal data

Nocturnal groaning syndrome is a common sleep disorder characterized by irregular groaning or vocalizations during nighttime sleep, representing a significant area of research in sleep disorders. Nocturnal groaning syndrome is a common sleep disorder characterized by irregular groaning or vocalizations during nighttime sleep, representing a significant area of research in sleep disorders. proposes a multimodal recognition approach based on speech, image, and text modalities. The study analyzes audio features using Mel Frequency Cepstral Coefficients (MFCC), which is the most common method for identifying nocturnal groaning syndrome. Coefficients (MFCC), extracts image features with pretrained MobileNetV2, and identifies key physiological signals from text using TF-IDF algorithm. Subsequently, Multimodal Compact Bilinear Pooling (MCB) is employed to fuse audio and image features, and a Text-Image CNN is used to combine image and text features. Support Vector Machine (SVM) is then used to classify the fused multimodal features, and decision-level fusion is performed using weighting criteria. Experimental results demonstrate an identification accuracy of 89.5% on the test set, significantly enhancing the auxiliary diagnostic effectiveness of nocturnurnal diagnosis. Experimental results demonstrate an identification accuracy of 89.5% on the test set, significantly enhancing the auxiliary diagnostic effectiveness of nocturnal groaning syndrome.