Development of an intelligent assistant model for environmental sound recognition for deaf people

Hearing is a key channel for perceiving the environment: a person recognizes a threat, reacts to everyday events, and maintains social contacts based on a sound signal. More than 430 million people in the world live with severe hearing loss, which makes them vulnerable to dangers. The development of mobile computing platforms and the availability of multi-channel microphones open up the possibility of creating a personal assistant that analyzes the acoustic scene in real time and translates sounds into vibration or visual alerts. The main challenge in building such a system is the high variability of sounds, the presence of strong background noise, and strict limitations on the computing resources of a portable device. To overcome these limitations, a hybrid neural network architecture is proposed in this paper, combining convolutional layers and bidirectional LSTM modules with an attention mechanism. The lack of representative Russian-language data was compensated by creating an extended corpus: the author's dataset "House-Office-Street" was added to the open sets UrbanSound8K and ESC-50, which in total gave 18,764 clips. Additional stability was provided by complex audio augmentation and generation of fragments with different signal-to-noise ratios. The result was a model with a volume of 4.8 million parameters, reaching an F1 measure of 0.852 and maintaining recognition accuracy at an SNR of up to –5 dB. The quantized mobile version provides an average latency of 128 ms with a power consumption of 178 mW on the Snapdragon 778G chipset. The developed prototype of the Android application generates reliable vibration and visual notifications, reducing the risk of accidents and increasing the social inclusion of people with hearing impairments.