In-bed gesture recognition to support the communication of people with Aphasia

People with language impairments can have difficulties expressing themselves to others, leading to major limitations to their safety, independence, and quality of life in general. Aphasia is an example of an acquired language impairment that affects many people (around 2 million in the United States), being commonly caused by stroke, but also by other brain injuries. Several augmentative and alternative communication solutions are available to help people with communication difficulties, but they are generally not suitable for all contexts of use (e.g., lying in bed). In the scope of the “APH-ALARM” project, which aimed at developing solutions to support people with Aphasia, we envision a system for the bedroom that enables conveying messages to be sent to a caregiver or relative, for example. Focusing on gesture input, in this contribution, we investigated if smartwatch sensors and machine learning (ML) can be used to recognise arm gestures executed while lying. We explored different factors, namely the feature set, size of the sliding window used for feature extraction, and ML classifier. The results obtained with data gathered from ten subjects are promising, with the best factor combinations for the user-independent solution leading to a mean macro F1 score of 94% or 95%. They demonstrate the potential of using wearables to develop a gesture input modality for the in-bed scenario, which can also potentially be extended to other contexts (e.g., sitting in a bed, chair, or sofa, or standing). This research also provides useful insights that inform future work, including the development and deployment of communication support systems that can benefit not only people with communication difficulties (e.g., more independence), but also those caring for them (e.g., more peace of mind). • A new gesture-based interaction modality for communication support is proposed. • The proposal is aligned with the needs of users with Aphasia while lying in bed. • The use of wearable sensors for arm gesture recognition is investigated. • The best achieved mean F1 score was of 95%, for a user-independent solution.