RoboSign: Improving Interaction Between Service-Robot and Hearing Impairments People

Service robots assist humans in routine activities such as domestic tasks. Their interaction with humans, e.g., during communication, is commonly done through voice recognition functionalities. However, voice interaction limits robot usage, making it inaccessible to hearing-impaired people. This paper presents RoboSign, an interaction solution between a service robot and people with hearing impairments through command recognition in Brazilian Sign language (Libras). RoboSign includes a simplified dataset from scratch for a computer with low computational power, training a recurrent neural network model using MediaPipe Landmark Detector and Long ShortTerm Memory (LSTM) to perform signal recognition. The model was validated during training with unknown data, reaching 99% average accuracy, and evaluated in real-time using a camera, reaching 85% average accuracy. These results demonstrate that it is possible to perform human-robot interaction through Libras recognition with considerable accuracy without high computational cost due to simplified data collection and the LSTM algorithm for network training.