I Hear, See, Speak & Do: Bringing Multimodal Information Processing to Intelligent Virtual Agents for Natural Human-AI Communication
In this demo paper, we present an Extended Reality (XR) framework providing a streamlined workflow for creating and interacting with intelligent virtual agents (IVAs) with multimodal information processing capabilities using commercially available artificial intelligence (AI) tools and cloud services such as large language and vision models. The system supports (i) the integration of high-quality, customizable virtual 3D human models for visual representations of IVAs and (ii) multimodal communication with generative AI-driven IVAs in immersive XR, featuring realistic human behavior simulations. Our demo showcases the enormous potential and vast design space of embodied IVAs for various XR applications.
Authors
- Lucie Kruse (ORCID: https://orcid.org/0000-0002-1914-7008)
- Erik Wolf (ORCID: https://orcid.org/0000-0003-4881-9982)
- Fariba Mostajeran (ORCID: https://orcid.org/0000-0002-7424-9031)
- Frank Steinicke (ORCID: https://orcid.org/0000-0001-9879-7414)
- Susanne Schmidt (ORCID: https://orcid.org/0000-0002-8162-7694)
- Sebastian Rings (ORCID: https://orcid.org/0000-0002-2020-4508)
- Michael Arz
- Ke Li (ORCID: https://orcid.org/0000-0003-0828-029X)
Institutions
- Universität Hamburg (DE)
- University of Canterbury (NZ)
- Human Computer Interaction (Switzerland) (CH)
Publication Details
- Published
- 2025-03-08
- DOI
- https://doi.org/10.1109/vrw66409.2025.00469
- Citations
- 2
- Primary Topic
- Speech and dialogue systems
- Type
- article
- Field-Weighted Citation Impact
- 4.96
Funders
- Ministry of Education
- HORIZON EUROPE Framework Programme