Will differences in keyboarding method of writing systems impact reading literacy performance: Insights from analyzing constructive response items in digital PIRLS 2021

Abstract The use of more digital devices in learning has motivated the transition from paper-based to computer-based formats in large-scale international assessments, with typewriting as the prevalent input method in such educational studies. However, there remains limited exploration on whether differences arising from varied encoding systems in keyboarding methods impact on students’ literacy performance. The current study, utilizing Item Response Theory, aims to investigate whether the use of different keyboarding methods has an effect on students’ reading performance based on the 2021 digital Progress of International Reading Literacy Study (PIRLS), with a particular focus on the constructive response items. The participating countries were divided into three groups based on the consistency between their test language’s writing system and its representations on the keyboard—with Group 1 having the highest consistency and Group 3 (Chinese is the only language in this group) the lowest. Results revealed that a significant keyboard effect exists for students with higher-than-average reading ability. Specifically, Group 3 scored lower than Groups 1 and 2, while Groups 1 and 2 did not exhibit significant differences in their reading performance. This suggests that for students with similar higher-than-average reading ability, typing in Chinese appears to be at a disadvantage compared to other input methods. This study emphasizes the importance of considering keyboard effects when designing and scoring future large-scale assessments in order to improve the overall validity of the tests.