Different but the same? Internal structure, measurement invariance, DIF, and reliability evidence across test modes

This study examined whether the validity and reliability of scores differed between online and face-to-face administrations of instruments that varied in length and measured typical response and maximum performance. A nonrandomized comparative desing was used. Typical-response data (5-item Perseverance Scale; 30-item Perceived Academic Self-Efficacy Scale) were collected from 568 middle school students (335 paper-and-pencil, 233 online); achievement data (9-item reading and 25-item listening tests) were collected from 94 university preparatory students (50 paper-and-pencil, 44 online). Model fit and measurement invariance were examined using confirmatory factor analysis and multiple-group confirmatory factor analysis for ordinal data respectively; differential item functioning (DIF) using Mantel-Haenszel and logistic regression, requiring statistical significance and a non-negligible effect size, and Cronbach’s alpha coefficients using Feldt’s F-test. Both typical response scales showed acceptable fit in both modes. The Perseverance Scale reached metric invariance while the Self-Efficacy Scale reached only configural invariance. Alpha did not differ for the Perseverance Scale (0.78 in both modes) or the long achievement test (0.80 online vs. 0.67 face-to-face, p = 0.097), but was significantly lower online for self-efficacy (0.83 vs. 0.88, p = 0.011) and lower face-to-face for the short achievement test (0.79 vs. 0.59, p = 0.027). No item met the DIF criterion in the short form; in the long form, none showed uniform DIF, but 3 of 25 items (12%) showed non-uniform DIF. Psychometric equivalence across modes could not be assumed and should be evaluated empirically for each instrument and population.

Authors

Institutions

Publication Details

Journal
Frontiers in Psychology
Published
2026-11-16
DOI
https://doi.org/10.3389/fpsyg.2026.1961544
Primary Topic
Psychometric Methodologies and Testing
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Different but the same? Internal structure, measurement invariance, DIF, and reliability evidence across test modes

Ömer Kamış, Neslıhan Tuğçe ÖZYETER, Hatice Özlem Anadol
Frontiers in Psychology
Psychometric Methodologies and Testing
article

Different but the same? Internal structure, measurement invariance, DIF, and reliability evidence across test modes

Ömer Kamış, Neslıhan Tuğçe ÖZYETER, Hatice Özlem Anadol
article en

Abstract

This study examined whether the validity and reliability of scores differed between online and face-to-face administrations of instruments that varied in length and measured typical response and maximum performance. A nonrandomized comparative desing was used. Typical-response data (5-item Perseverance Scale; 30-item Perceived Academic Self-Efficacy Scale) were collected from 568 middle school students (335 paper-and-pencil, 233 online); achievement data (9-item reading and 25-item listening tests) were collected from 94 university preparatory students (50 paper-and-pencil, 44 online). Model fit and measurement invariance were examined using confirmatory factor analysis and multiple-group confirmatory factor analysis for ordinal data respectively; differential item functioning (DIF) using Mantel-Haenszel and logistic regression, requiring statistical significance and a non-negligible effect size, and Cronbach’s alpha coefficients using Feldt’s F-test. Both typical response scales showed acceptable fit in both modes. The Perseverance Scale reached metric invariance while the Self-Efficacy Scale reached only configural invariance. Alpha did not differ for the Perseverance Scale (0.78 in both modes) or the long achievement test (0.80 online vs. 0.67 face-to-face, p = 0.097), but was significantly lower online for self-efficacy (0.83 vs. 0.88, p = 0.011) and lower face-to-face for the short achievement test (0.79 vs. 0.59, p = 0.027). No item met the DIF criterion in the short form; in the long form, none showed uniform DIF, but 3 of 25 items (12%) showed non-uniform DIF. Psychometric equivalence across modes could not be assumed and should be evaluated empirically for each instrument and population.

Frontiers in PsychologyVol. 17
Çankırı Karatekin University (TR), Kocaeli Üniversitesi (TR), Gazi University (TR)
Quality education
Psychometric Methodologies and Testing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon