Evaluating model generalizability for suicide attempt risk prediction: traditional machine vs deep learning

Suicide remains a leading cause of death and a significant public health concern in the United States. A majority (83%) of suicide decedents had a healthcare visit within the prior 365 days, presenting unique opportunities to utilize healthcare data for AI-based interventions. While previous works applied machine learning (ML) to analyze healthcare records for suicide attempt risk prediction (SARP), they lack external validation. Additionally, advantages of deep learning (DL) over ML for tabular SARP remains understudied. We performed external validation of a state-of-the-art SARP model from the Mental Health Research Network using over 750,000 UMass Memorial Health patient encounters. We further compared ML vs DL, assessing cross-setting healthcare generalizability. We found existing models did not generalize well, ML significantly outperformed DL on most metrics, and DL achieved higher sensitivity. These findings underscore the need for developing robust, generalizable SARP models for diverse healthcare contexts, improving identification of individuals at risk.