Making Social Interactions Accessible for the Blind and Low-Vision People by Detecting Emotions through Vocal Tone, Facial Expressions, and Body Language

Despite the thriving research in the field of accessibility, there are several niche area where there is a need for specific solutions. One such area is social interaction. Blind and Low-Vision people (BLV) are limited in identifying emotions through verbal communication alone, which is often biased and unreliable in certain environments. Prior research illustrates the importance of body language in expressions and articulates the mechanism behind human emotions. Drawing inspiratoin from such research and various studies that were focused on emotion recognition systems, our research focused on identifying a portable accessible solution for BLV that classified emotion based on multi-modal input such as facial expressions, body language and tone, with an audio feedback. 2 studies were conducted to understand the effectiveness of using an external aid in identifying emotions with 10 participants who identified as Low-Vision between the ages 31-45. Study one targeted at measuring the success of unassisted identification and Study 2 tested the effectiveness of prototype. BLV participants resonated with the purpose of the prototype and identified certain aspects that could be rendered in a different way. Although BLV were able to successfully detect certain emotions without assistance, most emotions, accompanied with background noise were harder to detect only based on tonality. Overall, the function of the prototype was identified to have a positive impact with minor limitations on the interface and feedback mechanics. As a part of our research we also proposed design considerations that can cater to the utility of such devices in multiple environments and thrive in conditions with multiple speakers. Further, we also proposed a co-design structure that can bestow ownership and enable active participation from BLV.