Inter-trial intervals bias reinforcement learning in humans

Although we know humans attempt to maximize reward per unit time when foraging, it remains unclear how the common practice of varying waiting time via inter-trial intervals (ITI) impacts reinforcement learning. Here, we tested several hypotheses concerning how time to wait during the ITI impacts reinforcement learning, including replay of recent outcomes, sensory decay of action representations, and discounting of action values due to added time costs. To test which of these accounts explains how ITIs impact reinforcement learning, we conducted a series of bandit-style tasks in which the ITI was either 1 or 4 seconds, expecting longer ITIs to allow replay that strengthens recent learning. Instead, across four studies (n = 96, 96, 439 and 117), a longer ITI made people more likely to change their last choice selectively after wins but not after losses. To explain why this effect occurs, we used computational modeling to demonstrate how the ITI cost competes with learned value for control of the next choice, with time costs impacting choice more when learned values were more uncertain. The effect of ITIs on choice generalized to a public fMRI dataset with jittered ITIs, and was absent in language-model agents playing our task. Taken together, our results suggest that ITIs bias human reinforcement learning, where people try to reduce uncontrollable time costs through switching to a new choice.