Dynamic hand exercise recognition for game-based finger rehabilitation

Frequent and intense exercise is crucial for rehabilitation, but motivation is often a barrier. Recent studies indicate that exergames can enhance motivation and exercise intensity by using targeted motor function for game control. Generally, one way to achieve this is to enable the control of such games by the targeted motor function. For example, in finger rehabilitation, selected exercises control exergames, relying on both representative hand gestures and accurate recognition systems. However, existing recognition systems in the literature are modeled based on hand gesture datasets that are not representative of common finger rehabilitation exercises. Therefore, this work deviates from the pull of previous works by developing a hand-gesture recognition system using a dataset collected specifically for the purpose of finger rehabilitation. The dataset comprises RGB images collected from 14 different subjects while performing 7 different finger exercises under varying backgrounds and lighting conditions. A learning network that leverages transfer learning of an off-the-shelf pre-trained VGG16 model using both feature extraction and fine-tuning is developed to recognize the seven different hand gestures featured in the dataset. The resulting models achieved an accuracy of 82.38% and 85.12% before and after fine-tuning respectively. Furthermore, the misclassification rate observed for specific classes was analyzed using the class activation map. The resulting model is integrated into an exergame framework and an experimental study conducted with 15 unimpaired participants demonstrates the suitability of the framework for game-based finger rehabilitation through user-experience measures such as Intrinsic Motivation Inventory (IMI), flow experience, and overall gaming experience.