Multiverse simulation to explore the impact of analytical choices on type I and type II errors in a reaction time study

Researcher degrees of freedom in data analysis present significant challenges in social sciences, where different analytical decisions can lead to varying conclusions. In this work, we propose an example of an exploratory multiverse simulation to empirically compare various decision pathways to identify an effect's sensitivity to different analytical choices. The approach is demonstrated on the congruency sequence effect (CSE), a well-studied phenomenon in cognitive control research. We reviewed existing literature to identify common non-theory-specific analytical decisions, such as outlier exclusion criteria and hypothesis testing methods, and incorporated these into our simulation framework. Using 20,000 simulated datasets, we compared the true positive rates (TPR) and false positive rates (FPR) across 50 different decision pathways, resulting in a total of 1 million analyses. Our results indicate substantial differences in power and type I error rates across the analytical pathways, with some posing a significant risk of producing high false positives. The findings underscore the importance of running extensive simulations to investigate different data handling and hypothesis testing approaches in certain research fields. This case study serves as an example for conducting similar simulation procedures in research fields characterized by high variability in analytical decisions when investigating an otherwise widely accepted effect.