Developing and validating a productive thinking assessment for sustainable university science learning

Abstract Assessing productive thinking in sustainable science learning requires instruments that capture coordinated cognitive and metacognitive processes during authentic problem-solving. Existing higher-order thinking assessments often examine critical thinking, creativity, or metacognition separately and provide limited validity evidence in performance-based sustainability contexts. This study developed and evaluated a performance-based Productive Thinking Assessment (PTA) measuring three interrelated dimensions: critical–analytical, creative thinking, and thinking process management within microplastic and bioplastic contexts. A multi-method instrument development design was used to obtain complementary validity evidence. Content validation involved seven experts, while psychometric evaluation involved 200 pre-service elementary science teachers from three Indonesian universities. Response-process evidence was examined using Think-Aloud Protocols with six students. Content validity was evaluated using Aiken’s V, internal structure using Principal Component Analysis (PCA) and PLS-SEM, item functioning using Rasch analysis, and scoring reliability using intraclass correlation coefficients. Aiken’s V coefficients ranged from 0.81 to 0.95. PCA supported the proposed three-dimensional structure, and PLS-SEM indicated high indicator reliability, internal consistency, and convergent validity. Rasch analysis showed acceptable item fit, with item difficulties ranging from − 0.829 to 1.289 logits and ordered, estimable rating-category thresholds. Inter-rater reliability ranged from moderate to good for individual ratings and from good to excellent for average ratings. The think-aloud findings illustrated evidence-based reasoning, solution generation, and metacognitive regulation during task completion. The findings provide complementary validity evidence supporting the interpretation of PTA scores for assessing productive thinking within sustainability-oriented science learning, while indicating the need for further validation across broader populations and science contexts.