Applying Multimodal Mixed Reality System for Classifying Parkinson’s Disease: Design and Evaluation of the Voice Module

Early detection and regular monitoring are crucial for the effective treatment and improvement of the quality of life in patients with Parkinson’s disease, a neurodegenerative disorder characterized by bradykinesia, tremors, muscle stiffness, balance issues, and speech impairments. Currently, diagnosis depends on frequent specialist visits, which can be burdensome for elderly patients. This work presents a prototype of a non-invasive tool for accurate Parkinson’s disease detection through voice analysis, utilizing a mixed reality head-mounted display. MR goggles (Microsoft HoloLens 2) offer a stress-free, in-home environment, minimizing the need for frequent hospital visits. Patients perform 5 voice-related tasks, and advanced speech recognition models (wav2vec2, wavLM, HuBERT) and text models (BERT) are employed for analysis. Data collection and system evaluation were conducted using the MR system, with 57 participants contributing to the dataset (including 21 patients diagnosed with Parkinson’s disease and 36 healthy controls). This technology holds great promise as an advanced diagnostic tool for neurodegenerative diseases, facilitating interactive assessments, reducing the strain on healthcare providers, and enhancing patient comfort by limiting the necessity of regular check-up visits.