Reflection-philosophy order effects and correlations across samples

Abstract Reflective reasoning often correlates with certain philosophical decisions, but it is often unclear whether reflection causes those decisions. So a pre-registered experiment assessed how reflective thinking relates to decisions about 10 thought experiments from epistemology, ethics, and philosophy of mind. Participants from the United States were recruited from Amazon Mechanical Turk, CloudResearch, Prolific, and a university. One participant source yielded up to 18 times as many low-quality respondents as the other three. Among remaining respondents, some prior correlations between reflective and philosophical thinking replicated. For example, reflection predicted denying that accidentally justified true beliefs count as knowledge. However, reflection test order did not impact philosophical decisions. Instead, a philosophical reflection effect emerged: making philosophical decisions before the reflection test improved reflection test performance. These and other data suggest causal paths between reflection and philosophy can go both directions, but detecting such results can depend on factors such as data quality.