Exploring Strategies to Improve Digital Literacy Assessment Using Log Data Analysis

The purpose of this study is to propose improvement strategies for national digital literacy assessment tools based on the analysis of log data from the first performance-based evaluation conducted in 2023. To achieve this, we analyzed log data from a total of 32,804 primary and middle school students. For the analysis, eleven types of data, including problem domain, problem type, and total number of logs, were utilized for the analysis. Students were assessed on their digital literacy level through 26 items, with primary school students given 40 minutes and middle school students given 45 minutes for the assessment. The key findings indicate that primary school learners generated 1.5 million log entries spanning four modules, whereas middle school participants produced 3.2 million log data points. Both primary and middle school students showed an increasing tendency to skip questions without answering as they progressed through the latter part of the assessment. Additionally, the tendency to skip questions increased when the minimum number of clicks required to solve a problem increased or when the problem length was longer. In the future, it is necessary to clearly define which parts of the log should be recorded in advance so that logs are consistently recorded. To accurately perform analyses such as student response type and pattern analysis, and error type analysis, a design for appropriate log data recording should be prioritized. This will enhance the reliability and validity of the tools and serve as a basis for future digital literacy policy development.