Enhancing audiovisual media experience with AI: insights from a pilot study on emotionally attuned subtitles for the d/Deaf and hard of hearing audiences

This article reports on a pilot study examining the role of emotionally expressive subtitles in improving the accessibility and inclusivity of audiovisual (AV) content for d/Deaf and Hard of Hearing audiences. Standard subtitling practices, although widely used, prioritize linguistic accuracy at the expense of conveying affective information crucial for narrative understanding. The study described here investigates alternative subtitling strategies that integrate visual design elements – such as dynamic typography, color, and motion - to reflect the emotional atmosphere of AV material. Drawing on multimodal theory and concepts from embodied cognition, the research explores how such subtitles may enrich a viewer’s experience by enhancing both comprehension and emotional engagement, particularly among audiences who rely on visual rather than auditory cues. Data were collected using an eye tracker and post-viewing questionnaires, which assessed participants’ perceptions of accessibility, narrative clarity, and affective resonance. While analysis of biometric measures remains ongoing, preliminary findings indicate that visually enriched subtitles may offer added value beyond conventional captioning methods. The study adds to growing interdisciplinary discourse on media accessibility, suggesting that emotionally attuned subtitling holds promise as a multimodal enhancement to traditional approaches, with implications for both practice and further research.