Rating procedures to evaluate generic critical online reasoning in an open Internet environment

Abstract University students regularly rely on Internet resources for academic work, requiring the ability to critically evaluate search results, identify relevant information, and draw well-founded conclusions. We refer to these skills as critical online reasoning (COR). This paper presents methods to assess Generic Critical Online Reasoning (GEN-COR) in an open Internet environment, focusing on online information acquisition (OIA) and reasoning based on evidence and synthesis (REAS). Based on data from two tasks, we illustrate the principles of task construction and the development of criteria to evaluate task performance. Using data from 376 first-year students from different fields, we examine inter-rater reliability, indicator difficulty, and the factorial structure of OIA and REAS. Results show that human ratings reliably assess GEN-COR performance and that OIA and REAS are distinct, although surprisingly uncorrelated dimensions. Further research is needed to validate these findings across tasks and to integrate critical information evaluation (CIE) skills.