Artificial intelligence scoring attitudes: scale development and validation

Abstract The increasing role of Artificial Intelligence (AI) applications in education necessitates the development of a valid and reliable measurement tool that can assess students’ attitudes towards AI-based scoring systems. The purpose of this study is to develop a scale that measures students’ attitudes towards AI-based scoring systems in education and to test the validity and reliability of this scale. In the study, a literature review was conducted and expert opinion was consulted to develop the scale items. The first form of the scale was administered to 416 participants. The construct validity of the scale was examined using exploratory factor analysis (EFA) and rotation procedures. As a result of these procedures, a structure consisting of 12 items and two main factors (AI-SAS positive attitude and AI-SAS negative attitude) was determined. In the next step, a confirmatory factor analysis (CFA) was carried out on the data obtained from 441 participants. The results showed that the scale has robust construct validity. To test concurrent validity, comparisons with the General Attitudes Towards Artificial Intelligence Scale (GAAIS) and the AI Anxiety Scale (AI Anxiety) revealed significant relationships between the AI-SAS and these scales. In addition, measurement invariance was tested to ensure that the scale would measure consistently across different demographic groups. The results showed that the AI-SAS scale has a similar factor structure in different groups according to demographic variables such as gender, type of school, use of artificial intelligence in daily life, and can therefore be used in different subgroups. In conclusion, this study provides a reliable and valid scale to measure students’ attitudes towards AI-based scoring systems in education. This scale can be used as a tool for evaluating the impact of using AI in educational practice.