A Chinese sign language recognition system combining attention mechanism and acoustic sensing

In recent years, with the widespread popularity of smart devices and the rapid development of communication and artificial intelligence technologies, sign language gestures that can break the communication barriers between ordinary people and those with speech and hearing impairments have received much attention. However, existing human gesture recognition methods include wearable device-based, computer vision-based and Radio Frequency (RF) signal-based. These methods have problems of being difficult to deploy, violating user privacy, and being susceptible to ambient light. Compared with the above methods, using ultrasonic signals to sense sign language gestures has the advantages of not violating user privacy and not being affected by ambient light. For that purpose, we use the built-in speaker and microphone of a smartphone to send and receive ultrasonic signals to recognize sign language gestures. In order to recognize fine-grained sign language gestures, we calculate the Channel Impulse Response (CIR) induced by the sign language action as a sign language gesture special. After that, we compute first-order differences along the time dimension of the Channel Impulse Response matrix to eliminate static path interference. Finally, a convolutional neural network containing convolutional layers, spatial attention, and channel attention is passed in order to recognize sign language gestures. The experimental results show that the scheme has a recognition accuracy of 95.2% for 12 sign language interaction gestures.