Different but the same? Internal structure, measurement invariance, DIF, and reliability evidence across test modes
This study examined whether the validity and reliability of scores differed between online and face-to-face administrations of instruments that varied in length and measured typical response and maximum performance. A nonrandomized comparative desing was used. Typical-response data (5-item Perseverance Scale; 30-item Perceived Academic Self-Efficacy Scale) were collected from 568 middle school students (335 paper-and-pencil, 233 online); achievement data (9-item reading and 25-item listening tests) were collected from 94 university preparatory students (50 paper-and-pencil, 44 online). Model fit and measurement invariance were examined using confirmatory factor analysis and multiple-group confirmatory factor analysis for ordinal data respectively; differential item functioning (DIF) using Mantel-Haenszel and logistic regression, requiring statistical significance and a non-negligible effect size, and Cronbach’s alpha coefficients using Feldt’s F-test. Both typical response scales showed acceptable fit in both modes. The Perseverance Scale reached metric invariance while the Self-Efficacy Scale reached only configural invariance. Alpha did not differ for the Perseverance Scale (0.78 in both modes) or the long achievement test (0.80 online vs. 0.67 face-to-face, p = 0.097), but was significantly lower online for self-efficacy (0.83 vs. 0.88, p = 0.011) and lower face-to-face for the short achievement test (0.79 vs. 0.59, p = 0.027). No item met the DIF criterion in the short form; in the long form, none showed uniform DIF, but 3 of 25 items (12%) showed non-uniform DIF. Psychometric equivalence across modes could not be assumed and should be evaluated empirically for each instrument and population.
Authors
- Ömer Kamış (ORCID: https://orcid.org/0000-0003-0605-087X)
- Neslıhan Tuğçe ÖZYETER (ORCID: https://orcid.org/0000-0003-1558-1293)
- Hatice Özlem Anadol (ORCID: https://orcid.org/0000-0002-2881-9806)
Institutions
- Çankırı Karatekin University (TR)
- Kocaeli Üniversitesi (TR)
- Gazi University (TR)
Publication Details
- Journal
- Frontiers in Psychology
- Published
- 2026-11-16
- DOI
- https://doi.org/10.3389/fpsyg.2026.1961544
- Primary Topic
- Psychometric Methodologies and Testing
- Type
- article
- Field-Weighted Citation Impact
- 0.00