Modeling intrinsic motivation as reflective planning

Why do people seek to improve themselves? One explanation is that improvement is intrinsically rewarding. This can be formalized in reinforcement learning models by augmenting the reward function with intrinsic rewards (e.g., internally-generated improvement signals). In this paper, we develop an alternative explanation: the drive for improvement arises from planning in a state space that includes internal states (e.g., competence). Planning is therefore reflective in the sense that it considers the value of future internal states (e.g., "What could I accomplish in the future if I improve my competence?"). We formalize this idea as a sequential decision problem which we dub the reflective Markov Decision Process. The model captures qualitative patterns of skill development better than a range of alternative models that lack some of its components. Importantly, it explains these patterns without appealing to intrinsic rewards.