Towards an RGB camera-based live repetition counter using auto correlation with action recognition for home rehabilitation

Patients recovering from stroke often struggle with their rehabilitation exercises at home, with personal therapists being both costly and potentially unavailable. Virtual rehabilitation programs may assist the patients by providing quantitative and qualitative feedback. The critical step for these automated systems is identifying and segmenting the performed activities for repetitions, making repetition counting an essential component. For this, an affordable, robust and generalized system is required. Previous works utilize expensive systems like Vicon and Kinect (discontinued), or require extensive training and are incapable of live counting. In this study, we propose a live repetition counter that works on RGB videos with Mediapipe, used for joint extraction. Also, we utilized action recognition (12 activities) using a lightweight transformer-based model. We employed a four-axis variance mechanism to monitor the motion across any arbitrary axis combined with the autocorrelation method to identify repetitions across random patterns. Our system was evaluated on a custom RGB dataset as well as the benchmark datasets UI-PRMD and KIMORE, achieving mean MAE values of 3.2733 and 1.4467, respectively, along with real-time validation through preliminary experiments. For activity recognition, the model achieved F1-score value up to 0.996. Relying solely on the RGB camera, our approach ensures practicality for home rehabilitation.