Explainable Reinforcement Learning via Causal Model Considering Agent's Intention

Explaining agent's decision can offer valuable insights for designers and end-users. One proposed method for explaining an agent's decision-making involves representing a causal relation between state components and action as a causal model and providing explanations for the decisions made using causal model. However, traditional causal model often faces structural limitation, restricting the range of representable control problems. Additionally, providing accurate explanation becomes challenging in control problems with various types of rewards because agent's intention of an action is unknown. In this study, we introduce a causal model capable of representing a broader range of control problems and a method to provide accurate explanations in control problems with various types of reward structures. Through redefining the relationships between nodes in the causal model, we have enabled a broader representation of control problems. Also, by incorporating agent's intention into the explanation, we have achieved to provide a more precise explanation. To validate the effectiveness of our proposed method, we conducted experiments using OpenAI's LunarLander environment. Using a proposed causal model, we defined the causal model of LunarLander, which could not be represented by conventional causal models. Furthermore, by incorporating the intentions of an agent into the explanation, novel interpretations previously inaccessible have become feasible.