Immersive Live Concert: A Multi-Sensory Experience Based on Real-Time Lyrics Detection from Spatial Audio Data

This paper presents a multi-sensory solution aimed at enhancing the immersive experience of remote audiences for a live blues concerts. Using the XRBLUES pilot of the HEAT project as a case study, the approach synchronizes olfactory and visual stimuli for remote viewers. The solution focuses on audio-based detection to identify key moments in the concert’s lyrics that trigger specific scents, such as ocean air and wind effects. These sensory cues are synchronized in real-time with the live concert, delivered via scent dispensers for remote users wearing XR headsets. Real-time communication is facilitated through an MQTT-based system to ensure minimal latency. The paper evaluates the effectiveness of this audio-based cue detection approach by evaluating Automatic Speech Recognition (ASR) models, using the Vosk Python library. This work demonstrates how scent-enhanced XR technologies can be integrated into live music events, offering a more engaging and immersive experience for remote concertgoers.