Detecting population-level concept drift in educational assessment data: A vulnerability gradient framework for LLM-era predictive validity monitoring

Educational Data Mining (EDM) pipelines assume that an online proxy (a continuous-assessment score) reliably predicts the latent construct it targets (autonomous competence) over time. Large Language Models (LLMs) provide a dateable exogenous shock against which this assumption can be tested. We frame the problem as population-level concept drift and propose a three-layer architecture to detect it: a distributional anomaly layer tracking skewness escalation and ceiling-score proportion; a predictive-validity layer tracking rank correlation between formative and summative scores; and a Vulnerability Gradient, a two-parameter typology (availability window, invigilation status) predicting drift severity. We validate it on a six-year case study of grade archives from an unproctored Moodle device (2020–2026, N = 4812), using the November 2022 ChatGPT release as a temporal breakpoint. The distributional layer detects severe post-breakpoint drift (mean shift +2.57/20, d = 0.82; skewness from −0.46 to −2.31; roughly a sixfold rise in ceiling scores). The predictive-validity layer reveals a non-monotonic rank-correlation trajectory (collapsing at partial adoption, ρ = .15; recovering at saturation, ρ = .70) even as the online–examination level gap grows monotonically, showing that rank validity and level validity fail differently and must be tracked separately. Given the observational design, findings are read as convergent evidence consistent with, not proof of, LLM-associated drift.