Empowering touchscreen accessibility for individuals with multiple sclerosis and dexterity impairments: a user-centered approach to touch-free gestures

Abstract Purpose Touchless interactions, like mid-air gestures, show promise in enhancing interaction and communication for individuals with physical disabilities. This study explores how mid-air gestural interaction can improve the accessibility of touchscreen devices, like tablets and other touchscreen tablet-like devices, for individuals with Multiple Sclerosis and dexterity impairments (MS-Dex). Methods Two studies were conducted. The first, assessed the challenges faced by ten MS-Dex participants with conventional tablet interactions. The second, adopting a user-centered design approach, explored mid-air gesture preferences among ten MS-Dex participants, aiming to develop alternative, more accessible interaction methods and to identify potential gestures for a touchless vocabulary. Gesture elicitation sessions were conducted, during which video recordings and Leap Motion Controller tracking were employed to record gesture metrics. This data was used to identify natural hand gestures suitable for 18 common tablet interaction scenarios. Both studies were compared holistically to understand the interplay between user challenges and gesture preferences, informing the development of effective and accessible touchless interaction solutions. Results The first study identified significant difficulties with activities requiring extensive movement and coordination. The second study showed that the elicited gestures were perceived as effective, intuitive, and easy to execute, confirming their accessibility for MS-Dex participants. Preference for gestures that minimized precise finger coordination and extensive hand movements was noted. Conclusion This research contributes to understanding the physical aspects of mid-air gestural interactions, suggesting design directions for more accessible tablet interfaces and providing insights into developing a standardized touchless gesture vocabulary, ultimately promoting greater accessibility for individuals with MS-Dex.