Meta-evaluating the Effects of Social Preferences on NPC-evaluators in an Energy Community Game

Energy Communities (EC) are emerging frameworks where citizens collectively share renewable energy.Levering knowledge about this topic is challenging for how varied these types of communities might be and how many actors are involved in decision taking.We are developing En-join, a game in which the player has to solve open-ended challenges that are mediated and evaluated by conversational agents that represent members of a EC.We implemented and prompted an LLM (Phi-4) to perform role-playing and evaluation simultaneously.We tested prompt variants indicating personality and behavior and meta-evaluated the evaluation performance using six predefined answers across three levels.Our results suggest that indicating social preferences noticeably affects the evaluation behavior.We contribute to the field of games and serious games by showing how LLMs can be used as conversational characters and evaluator agents simultaneously, and suggest that role-playing might be affecting evaluation behavior in any LLM implementations.