Multi-Modal Dysarthria Severity Assessment Using Dual-Branch Feature Decoupling Network and Mixed Expert Framework

Dysarthria, a motor speech disorder resulting from neurological damage, significantly impairs an individual's ability to articulate words. Assessing the severity of dysarthria is crucial for effective therapeutic interventions and rehabilitation strategies. However, current methods are time-consuming and subjective, leading to potential inconsistencies. This study proposed a dual-branch feature decoupling network and mixed expert framework for the automatic diagnosis and assessment of dysarthria. The proposed hybrid network architecture integrates a ResNet-based branch for audio data with an attention module and a 3D ResNet branch for video data, and was evaluated on the Multimodal Dysarthria Severity Assessment Challenge (MDSA) dataset, achieving significant performance improvements and ranking first in the challenge with a best score of 8.7404. The study's findings highlight the efficacy of the proposed framework in capturing essential features from multimodal data, contributing to more accurate and reliable assessments of dysarthria.