CNN-LSTM Proposal for Colombian Sign Language Greetings Classification: Preliminary Results

Colombian sign language (CSL) have shown different advances to automatize the translation and communication between deaf or disabling hearing people and speakers. However, these current efforts are oriented on classification of static signs, mainly. The present proposal implements the use a convolutional neural network (CNN) that also involves the use an architecture of the long short-term memory (LSTM) one. This allowed to include the time dependent information for dynamic signs like a sequence data. For this, a dataset was created with greetings for the CSL through the recording videos of signs. The methodology used the MediaPipe tool for feature extraction, which was used to represent relevant points from fingers articulation for frames of the videos. CNN-LSTM was applied with the joint points information for the classification. Results show that 71.50 % (+/-4.33%) for five greetings classification, employing a crossvalidation and early stopping criteria for the training of models.