AI Agents: Design and Evaluation of Gamified Conversational Agents

Effective social communication relies on understanding emotions, interpreting social cues, and maintaining meaningful conversations. To support the development of these skills, we propose a serious game featuring Conversational Artificial Intelligence agents that enable users to engage in questdriven conversations across various scenarios. Users can design custom scenarios and set specific conversation goals as quests to complete one-on-one conversations with a virtual agent who can express basic emotions. To provide structured guidance, the system offers predefined scenarios such as a date simulation, job interview, and Who Am I? game. In addition, it features Theory of Mind training to help people improve their ability to detect sarcasm, irony, and navigate everyday interactions more effectively. We design and evaluate our system to ensure a highly effective and engaging experience. Built on GPT-4o, our approach was evaluated using the Bot Usability Scale, with 20 users successfully recruited online. The results show that the agent's functionality was well received, showcasing high usability and effective communication. However, there is room for improvement in contextual relevance. Although LLM maintained coherent conversations, we faced challenges with long-term context retention. Furthermore, although the avatar's emotional expressions were generally accurate, further refinement is needed to better align them with the context.