Audio-Visual LLM for Augmenting Accessibility of 360° Video

Creators of 360° videos utilize affluent non-speech sounds for providing immersive experiences. The sound accessibility of such videos is essential for viewers, especially for d/Deaf and hard-of-hearing (DHH) people. In this paper, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AVLLM-360</i>, a multimodal framework using Large Language Models (LLMs) for understanding panorama video content and providing sound descriptions, which goes beyond the simple recognition of sound types. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AVLLM-360</i> integrates both visual and auditory information and bootstraps the cross-modal training from the pre-trained LLM. We also implemented a mixed-media interface that allows users to visualize the generated results hierarchically, enabling personalized customization of sound description generation when watching 360° videos. We conducted extensive experiments to evaluate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AVLLM-360</i>’s ability across a range of video understanding tasks. We also conducted qualitative studies with 12 DHH participants, evaluating the effectiveness of our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AVLLM-360</i> using 24 360° videos (covering different genres).