From framework to functionality: A cross-country analysis of PISA 2018 reading assessment framework’s item features as determinants of item difficulty

Abstract Background Theoretical frameworks excel in conceptualising reading literacy, yet their value hinges on their applicability for real-world purposes, such as assessment. By combining diverse theoretical frameworks, the Programme for International Student Assessment (PISA) 2018 designed an assessment framework for assessing the reading literacy of over 600,000 students globally. The PISA 2018 framework was operationalised into assessment items through construct-relevant item features aimed at determining item difficulty by mirroring the expectations set by the assessment framework. Methods Using cross-classified mixed effects models, we model the PISA 2018 reading item difficulty as a function of the PISA-defined construct-relevant and -irrelevant item features to gauge the extent to which item features determined item difficulty on average and across 71 countries. Results All five, delineated by the PISA framework, construct-relevant item features combined only reduce the variance in item difficulty by an average of 20%, with ‘Text Type’ being the only promising predictor. However, the construct-irrelevant ‘Item Format’ alone has half the predictive value of all construct-relevant features combined (10%), implying that external factors may unduly influence student outcomes. Conclusions These findings suggest that the functionality of item features within the framework leaves much to be desired, pointing to considerable shortcomings that necessitate a re-evaluation of the framework and its more effective implementation. We discuss implications for future PISA design and other stakeholders and call for bridging the gap between the PISA reading assessment framework and its practical application to ensure its validity, effectiveness, and credibility among educational stakeholders worldwide.