Developing and refining a scientific practices instrument for secondary science: methodological considerations

Assessing students’ scientific practices requires tests of reasoning underlying each practice. This study details the methodological considerations in developing a written-format scientific practices instrument for lower secondary students in Singapore, guided by the assessment triangle framework. The instrument targets six scientific practices across the investigating, explaining, and evaluating spheres of scientific activity using open-ended items to elicit reasoning. Item-specific rubrics distinguished performances across four levels. A pilot study with 82 eighth graders evaluated the instrument’s validity and reliability. Rasch analyses served as an item-level diagnostic tool, identifying problematic items infit/outfit statistics, person–item map, and category functioning. Examination of these items revealed tensions among construct alignment, item design, and scoring interpretations prompting revisions such as reclassifying targeted practices, redesigning items to foreground intended reasoning, and refining rubrics, while retaining high-demand items to preserve construct coverage. Findings highlight how instrument validity could be enhanced through careful operationalization of constructs, items that make reasoning visible, and discriminating rubrics. By making explicit these methodological decisions, this pilot study contributes a transparent account of the considerations in constructing a scientific practices instrument, with implications for researchers undertaking similar instrument development in science education contexts.