Personalized Cloud-Based sEMG-to-Haptic System to Enable Long-Distance Communication for Usher Syndrome

Abstract Usher Syndrome is a congenital disease that causes severe hearing and vision loss and is the leading cause of childhood deafblindness in the United States. Current assistive technologies are limited, expensive, and do not support long-distance communication. This paper presents a personalized, wearable, long-distance communication system for the deafblind. This system is cloud-based and presents a surface electromyography (sEMG) gesture classification system that uses personalized, small-scale Convolutional Neural Networks (CNNs) tailored to individual users, addressing the variability in muscle signals that hinders the adoption of sEMG-based systems. By creating user-specific sensor and gesture configurations, the system adapts to unique signal patterns, variations in electrode placement, and muscle geometries, making it highly robust to user-specific differences. It transmits real-time muscle activation data to a centralized server for model training, inference, and gesture recognition. Experimental results demonstrate that personalized models(~1 MB per model) significantly outperform cross-user models in classification accuracy, with an average improvement of 4.49% in validation accuracy over cross-user models in a 36-user dataset. By enabling low latency and minimal hardware requirements communication, this system is suitable for assistive communication technologies, including a novel sEMG-to-haptic sleeve that facilitates long-distance tactile communication for the deafblind. Moreover, the personalized model and cloud-based system approach is extensible beyond sEMG, offering a general framework for personalized sensor signal classification and paving the way for more accessible and effective solutions for the deafblind community and beyond.