Leakage controlled evaluation of machine learning for connectivity disruption in online examinations

Abstract Online examinations assume a stable connection; where it fails, a recorded score may reflect the network and not the learner. Machine learning seems an obvious way to tell the two apart, and high accuracies are easy to report. We ask how much of that accuracy is real. Using 675 examination sessions from 226 students in a connectivity-constrained setting, we audit our own released pipeline and then test what can be predicted without leakage. A naive protocol that oversamples before splitting and builds features from each student's whole history reaches 95.5–96.6% five-class accuracy with random forest, XGBoost and LightGBM. Under student-grouped folds, in-fold oversampling and prior-only features they reach 76.4–82.8%, below the 86.4% of always predicting the majority class. Detecting disruption from features of the same session is circular, because the label is a rule on those features: the answered share alone gives an AUC of 1.00. In the one non-circular task, predicting disruption before a sitting, examination type alone gives an AUC of 0.757 and adding the student's own history gives 0.789, a gain in AUC that a student-level bootstrap cannot separate from zero. Within practical sittings, students with and without a previous disruption were disrupted at the same rate (20.6% and 21.1%). Disruption follows the sitting and not the student. A three-tier triage of the predicted risk never isolates a group precise enough for automatic individual action. The results support context-level safeguards before a sitting and session-level review after it, and caution against profiling students as repeatedly vulnerable.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-11721007/v1
Primary Topic
Online Learning and Analytics
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Leakage controlled evaluation of machine learning for connectivity disruption in online examinations

Aiman A. AbuSamra, Samer Yaghi
1 links
Research Square
Online Learning and Analytics
preprint

Leakage controlled evaluation of machine learning for connectivity disruption in online examinations

Aiman A. AbuSamra, Samer Yaghi
preprint en

Abstract

Abstract Online examinations assume a stable connection; where it fails, a recorded score may reflect the network and not the learner. Machine learning seems an obvious way to tell the two apart, and high accuracies are easy to report. We ask how much of that accuracy is real. Using 675 examination sessions from 226 students in a connectivity-constrained setting, we audit our own released pipeline and then test what can be predicted without leakage. A naive protocol that oversamples before splitting and builds features from each student's whole history reaches 95.5–96.6% five-class accuracy with random forest, XGBoost and LightGBM. Under student-grouped folds, in-fold oversampling and prior-only features they reach 76.4–82.8%, below the 86.4% of always predicting the majority class. Detecting disruption from features of the same session is circular, because the label is a rule on those features: the answered share alone gives an AUC of 1.00. In the one non-circular task, predicting disruption before a sitting, examination type alone gives an AUC of 0.757 and adding the student's own history gives 0.789, a gain in AUC that a student-level bootstrap cannot separate from zero. Within practical sittings, students with and without a previous disruption were disrupted at the same rate (20.6% and 21.1%). Disruption follows the sitting and not the student. A three-tier triage of the predicted risk never isolates a group precise enough for automatic individual action. The results support context-level safeguards before a sitting and session-level review after it, and caution against profiling students as repeatedly vulnerable.

Research Square
Islamic University of Gaza (PS), University College of Applied Science (PS)
Quality education
Online Learning and Analytics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon