Multi-task Learning with Pre-trained Language Models for Mental Illness Screening

Major depressive disorder (MDD) and post-traumatic stress disorder (PTSD) are mental disorders that reduce quality of life. As they are challenging to detect in a timely manner, recent studies explore the mental illness screening potential of language models on modalities such as transcripts. Such datasets suffer from a limited number of participants. To overcome these challenges, we take a two-pronged approach: (1) we leverage multi-task learning to model multiple illnesses concurrently namely MDD and PTSD screening, and (2) we plug in pre-trained language models as a backbone, namely, Bidirectional Encoder Representations from Transformers (BERT) variants – which are capable of learning the linguistic content of clinical interview transcripts. In particular, we experiment with three multi-task weighting strategies and five BERT variants, applying them to 15 transcript sets extracted from the Distress Analysis Interview Corpus. Our results indicate that leveraging multi-task learning, especially with meta-weighting, increases the screening performance compared to single-task learning. Our multi-task learning model strategies improved the F1 scores for all 15 datasets for both MDD and PTSD screening. Notably, multitask learning improved MDD screening ability by 20 percent with transcripts regarding ‘regret’ and achieved F1 of 0.89 and 0.82 respectively for MDD and PTSD screening with transcripts regarding ‘medical history’. Our findings may help researchers develop more effective mental illness screening models.