Methodological Flexibility in the Iowa Gambling Task Undermines Interpretability: A Meta-method Review

Methodological flexibility–researcher degrees of freedom in procedure and scoring–can contribute to poor measurement and low replicability of findings. To quantify methodological flexibility in the Iowa Gambling Task (IGT), a widely used measure of decision-making in clinical psychology, we conducted a meta-method review of a random sample of 100 studies. We found prevalent under-reporting and wide-spread modifications of procedural details. We identified 244 distinct scores used to quantify IGT outcomes, 177 of which appeared only once in our sample. To quantify the potential impact of this flexibility, we performed a multiverse analysis using 205 scores on an open IGT dataset. Correlations among these ranged from rs = -0.942 to 0.998 with a median r = 0.022, suggesting that different scoring strategies can produce extremely different study outcomes. Given these findings, without substantive information about which specific procedural and scoring choices were employed, results from the IGT are rendered uninterpretable. Supplementary Materials, Data, and Code are available at github.com/anniria/igt-methodological-flexibility.