HiSTAG: Hierarchical Spatio-Temporal Attention over Part Groupings for Isolated Sign Language Recognition

Isolated sign language recognition maps a video of a single sign to its label; skeleton-based models reduce input videos from pixel data to temporal sequences of body and hand keypoints. This work presents HiSTAG (Hierarchical Spatio-Temporal Attention over Part Groupings), a three-stage skeleton encoder built on slots, each holding one body part within a short time window. Windowed graph attention, pretrained without labels by reconstructing masked keypoints, first encodes each slot; pooling attention then condenses each slot separately, and the slots of a window are merged into a single token before a temporal pooling cascade relates the windows. HiSTAG reaches 94.62% top-1 accuracy (mean of two seeds) on the 2,002-gloss signer-independent FDMSE-ISL benchmark at 9.44 GFLOPs per clip. With the same pretrained encoder it reaches 98.96% on INCLUDE and 98.75% on LSA64 (HWGAT subject split), matching the best published result there. Post-training low-rank truncation removes a quarter of the parameters at a cost of 0.05 accuracy points. Status: submitted to IEEE FG 2027, under review. Version 1.0 of this record contains the preprint (HiSTAG_preprint.pdf), the source code (histag-code-v1.0.zip), the FDMSE-ISL checkpoint for seed 0 (histag_fdmse2002_seed0.pt, 94.592% top-1) and the self-supervised Stage-1 weights (histag_stage1_ssl.pt). The checkpoints for the second FDMSE-ISL seed, INCLUDE and LSA64 will be added in a later version of this record. Scores and SHA-256 checksums for the files in this version are in MODEL_CARD.md and SHA256SUMS.txt. See README.md inside the code archive for reproduction steps.