Improved Swin Transformer-based fine-grained human motion understanding approach driven by simulation-based augmentation for intangible cultural heritage in sports

Fine-grained human motion understanding for Intangible Cultural Heritage (ICH) in sports faces challenges such as high similarity in posture between motion categories and significant differences in temporal rhythms within categories. Existing spatiotemporal motion models suffer from insufficient feature discrimination under small sample conditions and severe degradation in generalization ability when migrating from the simulation domain to the real domain. This paper proposes an improved Swin Transformer-based fine-grained human motion understanding method driven by simulation-based data augmentation for ICH. The method reconstructs the 3D skeleton of ICH motions using the Unity3D simulation engine combined with the OptiTrack motion capture system, and generates multimodal simulation samples by applying random perturbations to lighting, viewpoint, and body shape parameters. Addressing the limitation of the original Swin Transformer window self-attention mechanism lacking inter-frame temporal awareness, a Temporal Difference Attention (TDA) module is embedded after each Stage attention layer. This module explicitly models the rhythmic motion changes of ICH motions through the differential response of feature maps from adjacent frames. A cross-domain alignment loss, \(\:{\text{L}}_{\text{MMD}}\) , based on Maximum Mean Discrepancy (MMD), is introduced at the top of the feature pyramid to force the convergence of high-level semantic feature distributions between the simulation and real domains, enabling cross-domain transferability of spatiotemporal representations. The overall framework is jointly trained end-to-end to complete closed-loop modeling from simulation generation to real-world action classification and stage localization. Experiments are conducted on the public datasets NTU RGB + D 120 (hereinafter referred to as “NTU”) and Kinetics-400 for system validation. The accuracy rates for fine-grained human action recognition reach 88.6% and 79.3%, respectively. The mean average precision (mAP) in cross-domain scenarios reaches 81.2%; the MMD distance converges to 0.038; the temporal intersection over union (tIoU) for action stage localization reaches 0.76. Further generalization analysis is conducted on the SICHA-Derived sports ICH action test dataset (covering 8 categories of traditional actions such as Tai Chi and martial arts routines, totaling 1,240 annotated videos) derived from NTU. The proposed method achieves a Top-1 accuracy of 82.4%, a 10.6% point improvement over the strongest baseline, validating the framework’s practicality in the sports ICH scenario. Notably, under extreme small-sample conditions with only a single real sample per class, the framework achieves 71.6% accuracy, substantially exceeding both the simulation-only variant at 52.4% and real-data-only fine-tuning at 18.3%, demonstrating the practical value of the simulation-based augmentation and cross-domain alignment mechanisms for real-world sports ICH scenarios where annotations are inherently scarce.