A Role-Aware Explainable AI Dashboard for At-Risk Student Classification: Design and Technical Evaluation on the Open University Learning Analytics Dataset

Predictive models built on learning management system (LMS) data can flag students at risk of failing or withdrawing, but most systems present risk scores without their reasoning and give every staff role the same view. This study designed, implemented and technically evaluated a role-aware explainable artificial intelligence (XAI) dashboard that delivers explanations from one prediction system to instructors, academic advisors and administrators at different levels of aggregation. Using design science research and the CommonKADS knowledge engineering methodology, three ensemble classifiers were evaluated on the Open University Learning Analytics Dataset (32,593 student-module-presentation observations) with four-fold cross-validation grouped by course presentation; SHAP and LIME formed the explanation layer, and the prototype was integrated with Moodle and Canvas. The XGBoost prototype, selected on recall, achieved out-of-fold recall of 0.909, precision of 0.866, F1-score of 0.887 and AUC of 0.962, with accuracy ranging from 0.846 to 0.916 across held-out presentations. Both explanation methods ranked assessments submitted first and shared seven of their top ten features, but the mean per-student top-5 overlap was only 0.635, supporting the display of both methods side by side. Because the features span the full presentation, the results describe retrospective classification; practitioner trust, calibration and fairness remain to be evaluated.

Authors

Publication Details

Journal
EdArXiv (OSF Preprints)
Published
2026-11-17
DOI
https://doi.org/10.35542/osf.io/z79md_v1
Primary Topic
Online Learning and Analytics
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

A Role-Aware Explainable AI Dashboard for At-Risk Student Classification: Design and Technical Evaluation on the Open University Learning Analytics Dataset

Zainab Shamsiya Usman, Daniel Larry
1 links
EdArXiv (OSF Preprints)
Online Learning and Analytics
preprint

A Role-Aware Explainable AI Dashboard for At-Risk Student Classification: Design and Technical Evaluation on the Open University Learning Analytics Dataset

Zainab Shamsiya Usman, Daniel Larry
preprint en

Abstract

Predictive models built on learning management system (LMS) data can flag students at risk of failing or withdrawing, but most systems present risk scores without their reasoning and give every staff role the same view. This study designed, implemented and technically evaluated a role-aware explainable artificial intelligence (XAI) dashboard that delivers explanations from one prediction system to instructors, academic advisors and administrators at different levels of aggregation. Using design science research and the CommonKADS knowledge engineering methodology, three ensemble classifiers were evaluated on the Open University Learning Analytics Dataset (32,593 student-module-presentation observations) with four-fold cross-validation grouped by course presentation; SHAP and LIME formed the explanation layer, and the prototype was integrated with Moodle and Canvas. The XGBoost prototype, selected on recall, achieved out-of-fold recall of 0.909, precision of 0.866, F1-score of 0.887 and AUC of 0.962, with accuracy ranging from 0.846 to 0.916 across held-out presentations. Both explanation methods ranked assessments submitted first and shared seven of their top ten features, but the mean per-student top-5 overlap was only 0.635, supporting the display of both methods side by side. Because the features span the full presentation, the results describe retrospective classification; practitioner trust, calibration and fairness remain to be evaluated.

EdArXiv (OSF Preprints)
Quality education
Online Learning and Analytics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon