Human Joint Coordinate Sequencing in Video-Based Thai Finger Spelling Recognition

This article presents a video-based Thai finger spelling recognition system that utilizes human joint coordinate sequencing combined with deep learning models. The system processes video frames to extract key joint coordinates using MediaPipe, focusing on both hand and body poses. These skeletal points are fed into four deep learning models: LSTM, GRU, Bi-LSTM, and Transformer, which capture temporal dependencies in gesture sequences. For one-stroke poses with 10 letters, the Transformer model achieved the highest performance with an in-sample accuracy of 98.4% and an out-of-sample accuracy of 86.5%. Similarly, for two-stroke poses with 14 letters, the Transformer reached an in-sample accuracy of 94.2% and out-of-sample accuracy of 82.4%, outperforming the other models in precision, recall, and F1-score. These results demonstrate the effectiveness of the Transformer model in recognizing dynamic and complex Thai finger spelling gestures, offering a robust solution for improving communication accessibility for the deaf community.