Class-specific data augmentation of an imbalanced dataset for signer-independent fingerspelling recognition

Background Signer-independent fingerspelling recognition (FSR) aims to convert hand configurations into text or speech while remaining robust to inter-signer variations. Although existing models perform well in signer-dependent settings, their accuracy drops significantly in signer-independent scenarios. A key challenge is dataset imbalance, including class imbalance and class-subject imbalance, where underrepresented classes or signer-specific samples contribute to misclassification, especially among visually similar handshapes. Methods This study proposed a class-specific data augmentation strategy to improve signer-independent FSR within the Massey dataset of static American Sign Language (ASL) images. To address class–subject imbalance, an initial brightness augmentation was applied, followed by targeted augmentation of frequently misclassified class groups (0, 2, 6, O, V, W, M, N, and T) to mitigate persistent misclassification patterns. A lightweight three-layer end-to-end convolutional neural network (CNN) was trained to extract discriminative spatial features across 36 alphanumeric classes. All experiments adopted a leave-one-subject-out cross-validation (LOSO-CV) to evaluate signer-independent recognition. Results The proposed strategy improved signer-independent FSR accuracy on the Massey dataset from a baseline of 83.60% to 88.22% through multiple class-specific augmentation techniques. This improvement was statistically significant ( p = 0.0198), indicating reduced misclassification among visually similar signs and improved robustness in a static-image signer-independent setting under constrained data conditions.