AI-Powered Gesture-based Communication System for Enhanced Monitoring of Mouth Cancer

Mouth cancer patients experience difficulties in communication as a consequence of speech impairments caused by surgical procedures, radiation therapy, or tumor progression. However, this limitation has inevitably prevented them from indicating basic needs and consequently generates frustration, late responses to the caregivers, and reduced quality of life. Writing boards or manual signaling are all inefficient because it takes caregivers to be present at all times and to be constantly interpreting the writing. To solve this problem, this paper presents an AI-enhanced gesture-based communication system for supported diagnosis of mouth cancer. Furthermore, the system makes use of CNNs and RNNs to accurately process real-time hand gesture inputs. The Twilio API is used to instantly notify caregivers with SMS and voice alerts for recognized gestures so there is no delay in reaching out to them. Second, a drowsiness and inactivity detection mechanism is integrated that generates emergency alerts when alerted of critical conditions. Experimental evaluation showed very good inverted accuracy performance on gesture recognition, and adaptive learning helps performance over time. However, the communication delays of conventional methods were compared with those of the proposed method, and it was proved that the proposed method was better in terms of efficiency in caregivers. Minor challenges such as lighting variations have little effect on recognition but continuous training helps improve system robustness. A solution powered by AI, this booster provides patients non-verbal mouth cancer patients with greater patient autonomy, safety, and quality of care, making it a scalable and effective method.