Ανάλυση χρονοσειρών νοηματικής γλώσσας με χρήση μεθόδων μηχανικής μάθησης

This Diploma Thesis focuses on improving sign language recognition using machine learning methods. The main goal is to improve the recognition of American Sign Language (ASL) signs by using information about body and hand movements extracted from videos. The work is based on an existing approach that uses geometric distances to describe hand poses and replaces the self-supervised Triplet Loss with a supervised method called Proxy Anchor Loss. The main idea is to use the sign labels during training so that similar poses belonging to the same sign are closer together, while poses belonging to different signs are separated. For this purpose, skeletal data are extracted from videos using MediaPipe Holistic, which detects 75 three-dimensional body and hand joints in each frame. Each video is converted into a sequence of 80 frames, with 225 coordinate values per frame, and the data are normalized before training. A neural network called the Distance Encoder is then trained using Proxy Anchor Loss to learn useful representations of hand poses. The resulting features are combined with the original skeletal coordinates and processed by a Transformer model, which learns how the movements change over time and predicts the corresponding sign. The experiments also examine different ways of combining these features with geometric representations based on Euclidean distance, Procrustes distance and Articulated Pose Distance (APD). In addition, the Hyperformer architecture is adapted to the dataset and tested with and without an additional Proxy Anchor Loss objective. The proposed methods are evaluated on three subsets of the WLASL dataset: WLASL-100, WLASL-300 and WLASL-2000, which contain 100, 300 and 2,000 sign classes respectively. Their performance is measured using Top-1, Top-5 and Top-10 accuracy and compared with the existing geometric approach and other sign language recognition methods. The results show that Proxy Anchor Loss can improve recognition performance, especially when the number of sign classes is relatively small. On WLASL-100, the fine-tuned Proxy Anchor encoder achieves 78.3% Top-1 accuracy, compared with 76.5% for the best geometric baseline. On WLASL-300, combining Proxy Anchor with the three geometric encoders achieves 64.8% Top-1 accuracy, compared with 63.5% for the corresponding baseline. On the more challenging WLASL-2000 dataset, the Proxy Anchor encoder does not outperform the geometric methods when used alone. However, combining it with the geometric representations improves Top-5 accuracy from 66.9% to 69.5% and Top-10 accuracy from 76.8% to 78.7%. The Hyperformer experiments also show improvements when Proxy Anchor Loss is added to the training process. Overall, the results show that Proxy Anchor Loss can be a useful addition to skeleton-based sign language recognition, both on its own and in combination with geometric features. However, the proposed approach still has limitations, as it does not use facial expressions or mouth movements and focuses on the right hand during encoder pretraining. Future work could include these additional features, improve the use of both hands and evaluate stronger models with more extensive experiments.