Measuring undergraduate evolution learning using the CANS: psychometric strengths and limitations

Abstract Background Evolution continues to be one of the most difficult biological topics to teach, warranting innovative pedagogical tools and assessment strategies for enhancing evolutionary instruction. A major advance in measuring the evolution knowledge of undergraduate students came with the development of the Conceptual Assessment of Natural Selections (CANS). In this study, we use the CANS to measure knowledge and learning of natural selection in a large ( N > 6000) sample of undergraduate students to expand upon prior validity testing of this instrument and advance knowledge of student evolutionary reasoning. We apply the Rasch measurement framework to examine if the CANS productively measures the intended construct and investigate the patterns of knowledge and learning about evolution among students with different backgrounds and demographic characteristics. Results While a unidimensional Rasch model demonstrated acceptable reliabilities and fit for most of the CANS items, some items showed problematic fit statistics and were resistant to instruction. The instrument items also did not span the full range of student abilities, which suggests relatively low measurement precision. Our large sample also allowed rigorous tests of multidimensionality, revealing the presence of multiple dimensions or constructs, some of which may not be intentional. These results generated specific item-level recommendations for improving this instrument. Using Rasch measures to examine learning patterns, we found that pre-test evolution knowledge was low but that there were high learning gains by the end of the course. However, some concept categories were found to be more difficult than others, suggesting the need for more attention to these areas by instructors. We also identified pre-test disparities in evolutionary knowledge by socially defined race and biological sex, yet students from all groups achieved comparable learning gains at the end of the course. Conclusion The CANS holds great potential to generate critical insights about student evolutionary reasoning and provide information about which instructional approaches most effectively mitigate the notable knowledge disparities among students. We leverage the findings of this study to propose tangible ways in which this instrument may be improved in order to better achieve both of these goals.