Evaluating Large Language Models through Communication Games: An Agent-Based Framework Using Werewolf in Unity

In this study, we explore the reasoning capabilities of Large Language Models (LLMs) within the context of the social communication game Werewolf, aiming to evaluate their performance in managing complex system states commonly found in computer games.Our agent architecture gathers data, refines them into detailed information, and plans actions based on this knowledge.To demonstrate the feasibility of using LLM based agents in computer games, we developed a simulation and evaluation tool using the Unity game engine.This software enables users to experiment with various LLMs and agent architectures and to measure model performance within the application.For evaluation, we tested three models: GPT-3.5 Turbo, Mistral-7B-OpenOrca, and Nous-Hermes-Llama2-13B.The results show that even smaller models can perform reasonably well in Werewolf.However, their error rate is significantly higher, highlighting the need for additional software modules or fine-tuning to improve their accuracy.