Incremental Value of Structured Metadata for Text Based Risk Triage in a Role Play Derived University Counselling Corpus

Abstract Automated triage systems for student mental-health support may combine free-text messages with demographic and intake fields, but the incremental value of structured data requires empirical testing. This study compared text-only, structured-only and combined logistic-regression classifiers using a role-play-derived university counselling corpus. The original corpus contained 673 scenarios and the expanded corpus 1,672 scenarios; complete-case analyses used 654 and 1,509 scenarios, respectively. Under five-fold shuffled cross-validation, text-only escalation ROC-AUC was 0.705 (original) and 0.805 (expanded), compared with 0.687 and 0.799 for combined models and 0.633 and 0.752 for structured-only models. Paired bootstrap and repeated-split analyses found no demonstrated advantage from adding structured variables, while structured-only performance remained lower than text-only performance. Problem-category indicators carried most of the structured signal; demographic fields contributed little. At a 0.5 threshold, text-only sensitivity was 62.8% and 66.8%, while Critical-class recall in the four-class model was 25.0% and 64.9%. Achieving 95% escalation sensitivity required flagging 87% and 78% of cases. Structured metadata therefore provided no consistent incremental benefit once message text was available. However, participant-level grouping was unavailable, the corpus was authored through role-play, and no model achieved performance suitable for autonomous safety-critical triage. Independent validation on authentic counselling text is required before operational use.