Facial Expression Recognition and Integration with External Devices: A Computer Vision-Based Alternative Approach to Empower Individuals with Physical Impairments in Restoring Access and Control

This research introduces a novel approach aimed at enabling individuals with physical impairments, particularly those with irreversible motor neuron diseases (MND) such as amyotrophic lateral sclerosis (ALS), spinal muscular atrophy (SMA) and primary lateral sclerosis (PLS) etc., to regain independent control over external devices. The proposed approach utilizes a computer vision (CV) system, empowering individuals to operate and manipulate external devices autonomously. A YOLOv4-tiny convolution neural network (CNN) running on a Jetson NANO microcomputer with camera was used to recognize facial expressions such as Tongue-out and Eyes-closed. TensorRT optimization was adopted to enhance the performance of the CV system. Once features were recognized, electronic signals were fired from Jetson NANO via input/output pins. Arduino Pro Micro was also included for picking electronic signals from the CV system and firing actuation signals to external devices to realize device control. The proposed system showed over 97% of accuracy in single activation and over 89% of accuracy in communication simulation using the Augmentative and Alternative Communication (AAC) application. Therefore, the system showed a promising assistive method to support users regaining the ability of independent control and communication.