Cognitive Load Driven Audio-Visual Speech Enhancement Using a Neuro-Fuzzy Inference Model and OpenMHA Platform

Traditional hearing aids primarily amplify sound but often overlook factors critical for effective speech comprehension, such as listening effort and visual cues like lip movements. This article introduces an innovative, context and cognition-aware audio-visual speech enhancement (AVSE) framework for hearing aids, integrating adaptive and efficient edge implementation to improve user experience. Our approach includes adaptive speech enhancement model selection using neuro-fuzzy inference based on listening effort, audio signal-tonoise ratio (SNR), and visual cues. This work comprises four key components: integrating AVSE models with openMHA; using custom Radio Frequency (RF) sensors to estimate cognitive load; applying neuro-fuzzy inference for efficient speech enhancement selection; and developing hardware-accelerated Deep Neural Network (DNN)-based AVSE implementations. The framework includes a lightweight, customizable AVSE model optimized for real-time, power-efficient edge performance. The performance is comprehensively evaluated using metrics like STOI, PESQ, HASPI, MOS, latency, and power consumption, providing an indepth analysis of trade-offs between speech intelligibility, perceptual quality, auditory perception, processing speed, and energy efficiency, with hardware-accelerated inference ensuring optimal performance. This research signifies a significant advance in AVSE technology, offering enhanced communication and a better quality of life for individuals with hearing impairments.