SkillNet: Human Actions Assessment via Human-AI Collaboration

Intelligent human motion analysis is essential for developing next-generation IoT and AR/VR systems that enable automated, interpretable, and fine-grained performance assessment. Motivated by the need for real-time, explainable, and transferable skill evaluation, we propose a wearable sensing framework to assess human performance by tracking skill progression and minimizing injury risk. We use live badminton gameplay and workout exercises as representative use cases, where motion dynamics, postural stability, and limb coordination are critical to success. Both activities demand optimal posture and synchronized limb movements, while improper actions or suboptimal technique can lead to decreased performance and higher injury susceptibility. We introduce SkillNet , a multi-task learning framework that extracts shared representations across all limbs while preserving limb-specific motion signatures. The architecture employs task-specific regressors to detect subtle inter-limb dissimilarities and distinctive traits, enabling collective inference in a body sensor network (BSN) environment. To holistically measure performance, we formulated a weighted performance indicator (PI) that fuses AI-driven scoring with domain-expert evaluations, providing a robust metric for both qualitative and quantitative assessment. We evaluate SkillNet on three diverse datasets B adminton A ctivity R ecognition (BAR), M ulti- M odalities D ataset of S ports (MMDOS), and D aily and S ports A ctivities (DSADS) capturing a broad spectrum of motion types and skill intensities. Results show that SkillNet achieves an R \({}^{2}\) score of 86% and a mean squared error of 0.0093 in performance prediction. The integrated AI–expert scoring mechanism improves baseline performance estimation by 14.95% , demonstrating the advantage of combining human expertise with automated analysis. We further benchmark inference time, memory usage, and power consumption of the SkillNet , validating its efficiency and feasibility for real-time, end-to-end task inference on resource-constrained embedded edge devices, Jetson Nano and Jetson Xavier NX platforms.