Evaluating Explainability Faithfulness and Shortcut Learning in AI-Based Mental Health Text Screening: A User-Level Evaluation Framework

Text-based machine learning is widely proposed for the early screening of depression and anxiety, and published models often report high classification scores. Accuracy alone, however, does not show whether a model relies on clinically meaningful language, whether its explanations describe what the model actually does, or whether reported performance survives evaluation on people the model has never seen. This paper proposes an evaluation framework that examines four dimensions together: (i) prediction performance under user-grouped evaluation, (ii) faithfulness and stability of SHAP and LIME explanations, (iii) sensitivity to nonclinical shortcut features, and (iv) the usefulness of explanations for human reviewers. The framework compares a TF-IDF logistic regression baseline, a gradient-boosted tree model and a fine-tuned transformer, and specifies deletion/insertion faithfulness tests, shortcut-ablation experiments, nested user-grouped validation, user-level statistical inference and pre-specified decision rules. This manuscript is a proposed framework and evaluation protocol. No experiments have been run, and no empirical results are reported. The contribution is a reproducible, falsifiable design that can be applied to any suitable labelled corpus, together with an explicit separation of established limitations from claims that a systematic review must still verify.