Lip Movement Recognition Based on Keypoint Vector using Deep Learning

Lip movement recognition plays an important role in the lip reading system where it determines the performance of the system. Many lip movement methods have been developed, one of which is by analyzing the movement of lip landmarks. In this paper, a feature extraction method based on lip landmarks will be proposed, which we call keypoint vectors. The coordinates of the lip landmarks are converted into vectors to be resistant to changes in scale, rotation, and translation that may occur in the data. The features will be used as training data for deep learning models to perform classification tasks. In testing, the features will be divided into normalized features and non-normalized features. The models used are 1DCNN-LSTM and 1DCNN-GRU. The evaluation was carried out by analyzing the confusion matrix obtained from the model predictions. The test results showed that the model was able to classify words satisfactorily. These results indicate that the keypoint vector feature has the potential to improve the performance of the lip reading system.