YOLOv8-GR: real time gesture recognition by improving of YOLOv8 algorithm

Gesture recognition is pivotal for enabling intuitive human-computer interaction in resource-constrained environments such as smart homes, wearable systems, and edge computing devices. Although YOLOv8 achieves a good balance between speed and accuracy in the field of object recognition, it still has some shortcomings in the application of gesture recognition. Its SPPF module lacks sufficient global receptive field modeling, its detection head underperforms in complex backgrounds and multi-scale scenarios, and its CIoU loss is prone to localization instability on low-contrast and small targets. To address these issues, this study proposes YOLOv8-GR, an enhanced real-time gesture recognition model based on YOLOv8. The model integrates a Large Separable Kernel Attention (LSKA) mechanism into the backbone to capture long-range dependencies with low computational overhead, and introduces a Dynamic Head (DyHead) to unify scale-aware, spatial-aware, and task-aware attention, thereby improving detection robustness across gesture sizes and environments. In addition, the CIoU loss is replaced with Extended IoU (EIoU), which enhances bounding box regression stability in challenging visual conditions. To support deployment on edge devices, structured pruning is performed using the LAMP strategy, compressing parameters by 76.1% and reducing GFLOPs by 66.7%, followed by TensorRT acceleration on Jetson Orin Nano. Experimental results on a custom 11-class gesture dataset demonstrate that YOLOv8-GR achieves 0.97 [email protected] and 0.708 [email protected]:0.95, outperforming the baseline YOLOv8 by 22.5 and 10.0% respectively. After pruning and fine-tuning, the model maintains negligible accuracy loss while achieving 24.7 FPS, confirming its suitability for accurate, low-latency, and real-time deployment in constrained computing environments.