Assessing bilingual historical narratives: validating historical literacy descriptors aligned with the Common European Framework of Reference

Abstract The assessment of historical narratives, particularly in bilingual education contexts, presents unique challenges due to the intricate interplay of communicative and disciplinary competences. This study aims to provide a quantitative validation for B1 and B2 level CEFR-aligned descriptors developed to measure historical literacy competence by analyzing the reliability of the measures obtained for the raters and for the properties and functioning of the rating scale in which they were integrated. The paper thus explores how quantitative analysis of rater-mediated assessment contexts where rating scales are applied can contribute to the construction of validity arguments, backing the usage of the assessment instrument. Text fragments were extracted from various corpora of historical essays written by students in CLIL secondary education programs and from university students’ productions and were rated using key descriptors of historical literacy competence integrated into a 4-point analytic rating scale to conduct a post hoc analysis of the rating process. For this purpose, the Many-Facet Rasch Model (MFRM) statistical analysis was employed to examine how raters interact with the descriptors while assessing students’ performances. Results showcase that raters can reliably interpret the descriptors, which can be used to successfully differentiate between various historical literacy competence levels. This study may provide a more solid backing for the incorporation of integrated assessment instruments in bi- and multilingual educational contexts, aiming to increase equity and fairness in assessment practices.