Do You Agree? Do You Strongly Agree? The Effect of the Number of Response Categories on Response Processes and Verification of Substantive Hypotheses

Abstract This study investigates how the number and labeling of response categories in survey scales affect respondent behavior, psychometric properties, and substantive conclusions. Using data from a web-based survey experiment with over 2,800 participants, we randomly assigned respondents to scales that varied in the number of response options and labeling formats. Engagement, response style tendencies, reliability and convergent validity were assessed through a combination of process data (e.g., response times, cursor movements), self-reports, and psychometric modeling. Our findings suggest that scales with a greater number of response categories tend to exhibit higher reliability, although the improvements are modest and depend on the specific scale. While scales with more response options required longer completion times and prompted more complex response behavior, the number of response categories did not influence substantive conclusions in simple regression models.