A segmentation method for synthetically-trained pose-based fingerspelling recognition models

This paper presents a novel sign language fingerspelling segmentation technique. We demonstrate that the "holds" or "apogees" in fingerspelling sequences are revealed by comparing the similarity between adjacent frame poses. After segmenting these "hold" frames, where the handshape pose is fully articulated, we can apply light-weight models to classify the fingerspelling letter. Two such models are presented here: a three-layer multi-layer perceptron (MLP) and a single-layer MLP-LSTM, both of which were trained on synthetic fingerspelling datasets of American Sign Language (ASL). We evaluate our approach using a dataset derived from "The ASL Fingerspelling Recognition Corpus (version 1.0)" (ASLFR) on the Kaggle platform, the largest publicly available repository of ASL fingerspelling pose data. Variation in signer fluency, dialect and signing speed make this corpus a challenging test dataset for the models. The fingerspelling recognition performance of the models demonstrates the potential of the segmentation technique and the use of synthetic training data to create resource-efficient pose-based sign language recognition (SLR) pipelines.