Multi-modal Interaction with Virtual Agents: A Pick-up and Placement Case Study

This paper presents a multi-modal human-agent interaction framework that integrates hand tracking, voice commands, user intention interpretation, and scene understanding using large language and vision models for object pick-up and placement tasks in VR. Scene understanding integrates object tags and ChatGPT-4V’s object detection. User commands are processed via the LLaMA-3.2 model, which interprets user intention, identifies task and reference objects, and spatial relationships for task execution. An initial experiment with 15 participants showed that the multimodal approach achieved the fastest task completion times and highest user satisfaction. While hand-only interaction was comparable, voice-only interaction was significantly slower.