Interpretable and Sample-Efficient Reinforcement Learning by Discovering Value-Based HTN Models

Despite great success across various domains, reinforcement learning (RL) faces significant challenges in sample efficiency and interpretability, particularly in sparse reward scenarios. To address these issues, researchers have explored integrating symbolic planning with RL, leveraging symbolic models to provide high-level guidance to RL agents. However, existing methods often rely on manually crafted knowledge or overlook hierarchical task structures, leading to suboptimal planning, ineffective option policy training and low sampling efficiency in complex environments. In this paper, we thus propose a novel RL framework that incorporates value-based hierarchical symbolic models learned directly from the environment. This framework establishes a loop training process to automatically construct value-based hierarchical task network (HTN) models, including action models and method models embedded with intrinsic value, derived from RL interaction trajectories. The learned hierarchical symbolic models alleviate the reliance on expert domain knowledge, provide interpretable and globally optimal plans to guide the training of low-level option policies, and significantly enhance sample efficiency. We empirically validate the interpretability of subtasks and demonstrate improved sample efficiency compared to state-of-the-art approaches across several domains.