Enhancing Visitor Engagement in Interactive Art Exhibitions with Visual-Enhanced Conversational Agents

Conversational agents in art exhibitions can enhance user engagement and understanding of artworks by providing contextual information, especially through voice interactions. However, creating a deeper personal connection with art - which often requires direct aesthetic and visual experiences - remains a challenge. This paper examines how integrating visual perception into conversational agents can enhance alignment with visitors' artistic interpretations, thereby fostering deeper engagement with interactive art exhibitions. We introduce a voice-based conversational agent enhanced with visual capabilities via a multimodal large language model (MLLM), allowing the agent to perceive, interpret and discuss artworks in real-time with visitors. The system utilizes a simplified Retrieval-Augmented Generation (RAG) architecture, which collects voice inputs, retrieves relevant information from a domain knowledge graph, and uses the LLM to generate conversational responses, which are then converted into voice outputs. A user study with 36 participants, divided into two groups, was conducted to compare the enhanced system with a baseline system that lacked visual input. Our results show that the visually enhanced system significantly improved visitor engagement and perception. Content analysis of the conversational transcripts further revealed a wider range of conversational topics, deeper visitor perceptions, and the agent's ability to provide more nuanced, visually-related discussions.