What do the Indonesian National Assessment public-release indices measure? A five-cohort evaluation of construct validity, response quality and cross-cohort comparability

Background The public microdata release (Rapor Publik) of the Indonesian National Assessment (Asesmen Nasional, AN) has become the principal secondary dataset in Indonesian educational research, yet its measurement properties have not been independently evaluated. Methods Five consecutive senior-secondary releases (2021–2025; N = 222,918 grade XI students in 1,137–1,265 schools) were harmonised and subjected to an identical diagnostic battery per release: convergent and discriminant correlations among the 21 character indices; criterion associations with achievement, adjusted for a general self-report factor and decomposed within and between schools; response-quality gradients; and cross-release stability. Consistency across waves, rather than significance, served as the evidential criterion. Results Indices sharing a construct label correlated no more strongly with one another than with unrelated constructs in every release. In our case, the creativity composite shared 27–51% of its variance with a general self-report factor, and its association with literacy reversed sign once that factor was controlled (partial r = −.09 to −.12). The school-mean general factor correlated.71–.84 with school-mean literacy. Book ownership correlated negatively with achievement throughout. Achievement metrics drifted with no documented linking; official composites were reconstructed at least twice; two rotated designs leave only 0.1% of the 2025 sample complete on all character indices; and all releases share the 2021 school frame, forming a panel rather than independent samples. Conclusions Several widely used AN indices do not behave as their labels imply, and no score is comparable across releases. Student-level secondary analyses require latent or adjusted measurement, multilevel specification, response-quality screening, within-release standardisation, and recognition that cross-release consistency is not independent replication. A harmonised dataset and full R pipeline are provided.