Sign Language Motion Recognition using Machine Learning and RGB Image Processing

In this paper, we propose a method to recognize sign language motions using machine learning and RGB image processing techniques. We considered 24 different sign language words and obtained 15 video samples per word, either from public datasets or by direct recording. OpenCV and MediaPipe were used to acquire the key points of the human body skeleton from each video frame, which were then used to build the training data for sign language motion rec- ognition. Two types of training data were used: raw data and feature data. The raw data con- sists of 225 position coordinates, and the feature data consists of 54 rotation- and scale-invariant values. Convolutional neural networks (CNNs) and multilayer LSTM networks were then trained on both types of data to recognize sign language motions, and their effectiveness was evaluated through experiments. The results demonstrated that using the feature data yielded higher accuracy and effectiveness than using the raw data. Additionally, multilayer LSTM net- works outperformed CNNs regardless of the type of data used.