Reliability-based arbitration reveals heterogeneous model-based and model-free reinforcement learning strategies linked to compulsivity and impulsivity

The mechanisms by which humans arbitrate between reinforcement-learning strategies remain poorly understood, and the nature of these strategies has come under recent debate. Using a modified two-step decision task in two independent samples (N = 452 and N = 226), here we show that humans integrate three forms of reliability information when arbitrating control: transition reliability, model-free reward-prediction reliability, and model-based reward-prediction reliability. These mechanisms are deployed differently across individuals, revealing predominantly model-free, model-based, mixed, and non-reinforcement learning behavioral profiles that are associated with distinct compulsivity and impulsivity profiles. Critically, individuals also differ in what they arbitrate over: some switch between model-based and model-free control, whereas others maintain model-based control while arbitrating between alternative task representations. Together, these findings reveal broad heterogeneity in human reinforcement learning and provide deeper insight into individual differences in cognition and neuropsychiatric vulnerability. People use a mixture of goal directed planning and automatic responses to make decisions in changing environments. Here, the authors show that humans use reliability signals to arbitrate between strategies, and that people differ in which strategies they arbitrate between.