Smartphone-Based Detection of Early Parkinson’s Disease With Tapping Records and a Multimodal-Multiscale Ensemble Network

Potential Parkinson’s disease (PD) patients have increased dramatically in the aging society, and it is necessary to explore an automated detection method for early PD patients. Smartphones, with extremely high ownership rates, instant communication, abundant sensors, and high-speed network access, provide opportunities for the daily detection of early PD. This article develops a multimodal-multiscale ensemble network (MMSENet) to detect early PD patients using smartphone-tapping records. Specifically, the acceleration data-converted images and preprocessed coordinate data are fed into cascaded multihead convolutional neural network (MH-CNN) blocks for feature extraction, followed by feature fusion and classification, with the final results based on both data types being integrated through a result ensemble module. The MH-CNN block consists of two convolutional heads and a residual connection to extract multiscale features. A novel intramodal feature fusion module based on attention mechanisms is designed to fuse features from different images. The results ensemble module considers the metamodel probability and mean probability to improve the robustness and reliability of the detection results. Experimental results demonstrate that MMSENet can accurately detect early PD patients, achieving an AUC of 0.995, an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$F1$ </tex-math></inline-formula> of 0.895, and an accuracy of 0.965.