Emotion Sonification As Opportunity For Inclusive Texting: Replacing Emoji With Vocal Bursts And Prosodic Voices

Sentiment in text messages is commonly emphasized through emoji. These pictograms are prone to misconception and non-inclusive of screen reader users due to the alt texts’ dissociation from visual emoji. Sonified emotions instead of emoji may provide an inclusive alternative. In this work, we sonified seven emotion categories relevant to text-messaging as prosodic text and vocal bursts. A quantitative online study with (n = 27) blind and visually impaired and (n = 19) sighted participants revealed that the combination of both voice qualities yields the highest recognition rates and most confident ratings. Best results are achieved if vocal bursts precede a prosodic voice. Our findings directly translate to practical implications for conveying sentiment in screen readers and voice assistants. It is up for future work to determine whether wrap-up effects known from visual processing of emoji in written text are replicable in acoustic processing of emotional cues.