Immersive Tailoring of Embodied Agents Using Large Language Models

LLM-based embodied agents are recently emerging in VR, supporting various scenarios such as pedagogical assistants, virtual companions, and NPCs for games. These agents hold potential to enhance user interactions but require thoughtful design to cater diverse user needs and contexts. We present an architecture that leverages different LLM modules to enable conversational interactions with an embodied agent in multi-user VR. Our system’s primary goal is to facilitate immersive tailoring through conversational input, allowing users to dynamically adjust an agent’s behavior and properties (e.g., role, personality, and appearance), directly within the virtual space, rather than during development or via separate interfaces. We evaluate the system’s performance, measuring latency during tailoring tasks, and share insights from a six-week study involving five users exploring various scenarios. While the approach shows promise, challenges remain, including reducing latency in the speech-to-text-to-speech pipeline and addressing the black-box limitations of LLMs, highlighting areas for future research.