Development and Validation of the AI Competency Scale for High School Students Using Scenario Method

This study aimed to develop and validate a measurement tool for assessing AI competencies for high school students using the scenario method. To this end, based on a comprehensive literature review, this study constructed five sub-domains of AI competencies: basic understanding of AI, AI-based problem-solving, AI-based research and development, AI-based evaluation, and AI ethics, and it also constructed three sub-elements for each sub-domain. The preliminary version of the AI competency measurement tool was developed with three scenarios for each sub-domain and three items for each scenario (a total of 15 scenarios and 45 items), which were revised and improved based on both a focus group interview with seven high school students and a review of content validity by ten experts in the field. Subsequently, a pilot test using a preliminary version of the tool was conducted with 221 high school students. Using the data, this study conducted confirmatory factor analysis and reliability analysis. Consequently, the finalized AI competency measurement tool for high school students (a total of 15 scenarios and 30 items) was good in terms of construct validity and reliability.