Learn to Detect Freezing of Gait in Parkinson's Disease Patients Using Mobile Sensors Assisted by Video Data

Freezing of gait (FoG) is a potentially dangerous symptom affecting a majority of patients with Parkinson's disease that can lead to injury and even death. Recent advances in mobile IoT sensors enable the monitoring of patient activity in real time to reduce the risk of FoG episodes. The abundance of sensor data suggests a deep learning approach to predicting FoG before they occur. In this work, we propose a new algorithmic framework for FoG detection that trains on both sensor data and associated video data. Using the richness of visual data during training, we are able to derive more accurate assessments when making predictions based solely on sensor data during deployment. This method is motivated by the two-tower models that have been utilized in recommender systems and in multimodal learning. The resulting models achieve higher performance (e.g. 80% accuracy) than models trained on sensor data alone (76% accuracy).