Development and validation of a scale to assess algorithmic literacy in the context of recommender systems (ALRS)

Algorithmic systems, particularly recommender systems, are central to online platforms yet remain largely opaque to users. Algorithmic literacy has therefore emerged as a critical competence to foster informed and autonomous engagement, especially among younger audiences. This study developed and validated a multidimensional scale, Algorithmic Literacy in the Context of Recommender Systems (ALRS), focused on online audiovisual entertainment platforms. An initial pool of 51 items, derived from the literature, was refined through expert content validation, cognitive interviews, and lexical review, resulting in 38 items. The instrument was tested with 1,564 Portuguese students aged 15–25. Reliability analyses showed strong internal consistency (α = .955; ω = .955; Spearman-Brown = .881). Exploratory factor analysis identified five dimensions, and confirmatory factor analysis on an independent sample supported a second-order structure with excellent fit indices (χ 2 /df = 1.55; CFI = .991; RMSEA = .040). Composite reliability and average variance extracted were also satisfactory. The final 29-item ALRS scale demonstrates robust psychometric properties across five dimensions—awareness, knowledge, assessment, reflection, and practical use—offering a valid and reliable instrument to assess algorithmic literacy in the context of recommender systems, with applications in education, research, and policy. • This study introduces the ALRS, a new multidimensional scale designed to assess algorithmic literacy specifically in the context of recommender systems on online entertainment platforms. • The scale was developed and validated using data from a representative sample of 1,564 Portuguese youth aged 15–25. • Exploratory and confirmatory factor analyses supported a five-factor structure—awareness, knowledge, assessment, reflection, and practical use—grouped under a second-order latent construct. • The ALRS demonstrated excellent internal consistency (α = .941; ω = .942), good split-half reliability (Spearman-Brown = .881), and excellent model fit (CFI = .991; RMSEA = .040). • The validated scale provides a robust instrument for researchers, educators, and policymakers to assess and promote critical engagement with algorithmic systems.