Developing and validating a multi-tiered instrument for assessing knowledge and reasoning about socioscientific issues

Abstract Today’s information-rich societies increasingly require citizens to evaluate scientific claims that carry social, ethical, and political consequences. Socioscientific issues in Earth and environmental science are particularly challenging because they involve complex evidence, competing perspectives, and unfortunate opportunities for misinformation. Meaningful engagement with these issues depends not just on accurate scientific knowledge but also on the quality of reasoning that is used to justify conclusions. We developed the Reasoning About Socioscientific Issues (RASSI) instrument, a 12-item, three-tiered assessment that is designed to examine how individuals’ knowledge of Earth and environmental socioscientific issues aligns with scientific consensus for each issue; how individuals reason about those claims; and their confidence in that reasoning. U.S. adults were recruited via Prolific ( N = 498) and we evaluated the instrument’s structure and reliability through a split-sample approach that included Exploratory (EFA) and Confirmatory Factor Analyses (CFA). EFA results supported a clear structure for each of the three tiers of the RASSI and good to excellent internal consistency across the tiers. The CFA supported a three-factor model with excellent relative fit (CFI = .96, TLI = .96), though RMSEA was elevated (RMSEA = .106). Reliability was good to excellent across tiers (ωs = .856–.919). Latent factors were positively correlated, especially knowledge and reasoning ( r = .96). A two-factor alternative combining knowledge and reasoning did not improve fit, supporting the three-factor model. Evidence toward establishing construct and face validity was also collected through think-aloud interviews and expert review. Overall, the RASSI instrument provides a practical, multi-tiered tool to measure socioscientific understanding in a way that integrates knowledge, reasoning orientation, and confidence. Implications for research and instruction in Earth and environmental science contexts are discussed.