Predicting suicide death among veterans after psychiatric hospitalization using transformer based models with social determinants and NLP

Predictions of suicide death of patients discharged from psychiatric hospitals (PDPH) can guide intervention efforts including intensive post-discharge case management programs, designed to reduce suicide risk among high-risk patients. This study aims to determine if additions of social and behavioral determinants of health (SBDH) as predictors could improve the prediction of suicide death of PDPH. We analyzed a cohort of 197,581 US Veterans discharged from 129 VHA psychiatric hospitals across the US between January 1, 2017, and July 1, 2019 with a total of 414,043 discharges. Predictive variables included administrative data and SBDH, the latter derived from unstructured clinical notes via a natural language processing (NLP) system and ICD codes, observed within a 365-day window prior to discharge. We evaluated the impact of SBDH on the predictive performance of two advanced models: an ensemble of traditional machine learning models and a transformer-based deep learning foundation model for electronic health records (TransformEHR). We measured sensitivity, positive predictive value (PPV), and area under the receiver operating characteristic curve (AUROC) overall and by gender. Calibration analysis was also conducted to measure model reliability. TransformEHR with SBDH achieved AUROC of 64.0 Specifically, ICD-based SBDH improved AUROC by 3.1% (95% CI, 1.6% - 4.5%) for the ensemble model and by 2.9% (95% CI, 0.5% - 5.4%) for TransformEHR, compared to models without SBDH. NLP-extracted SBDH further improved the AUROC: 1.7% (95% CI, 0.1%- 3.3%) for ensemble model and 1.8% (95% CI, 0.6%- 2.9%) for TransformEHR. TransformEHR achieved 0.2%, 0.4%, 0.8%, 1.6% PPV per 100 PDPH 7, 30, 90, 180 respectively. Moreover, TransformEHR showed superior calibration and fairness compared to ensemble model, with SBDH further improving fairness across both predictive models. In conclusion, both ICD-based SBDH and NLP-extracted SBDH improved the performance, calibration, and model fairness of prediction of suicide death for Veterans after their psychiatric discharge.