TFS Point-on-Hand Sign Recognition Using Part Affinity Fields

Our study investigates an application of a bottom-up design for keypoint regression, Part Affinity Fields (PAFs), for sign language recognition. Automatic sign language recognition could facilitate communication between deaf people and the hearing majority. Sign languages generally employ both semantic and finger-spelling signing. Semantic signing includes acting out to convey meaning, while finger spelling complements signing through the spelling out of proper names. Specifically, this article addresses an automatic recognition framework for the static point-on-hand (PoH) signing of Thai Finger Spelling (TFS)—the finger-spelling part of Thai Sign Language (TSL). From a pattern recognition perspective, PoH signing is quite distinct among signing schemes for requirement of precise localization of key parts on the signing hands. A recent study addressed PoH using an off-the-shelf version of MediaPipe Hands (MPH) and found shortcomings particularly when there was a high degree of hand-to-hand interaction. The top-down design of MPH was hypothesized to be the culprit. Our study investigates a bottom-up design, Part Affinity Fields (PAFs), along with examination of the related factors. The results support the hypothesis of a high-degree of hand-to-hand interaction posited by the MPH study. However, the overall performance of the PAF-based approach is shown to be modestly effective (72% accuracy vs. 58% and 47% of the MPH- and X-Pose-based approaches). In addition, its generalization is shown to be lacking. Thus TFS point-on-hand sign recognition remains a challenge.