Developing and Validating Language-Based Assessments for Mental Health : Measuring and Describing Depression, Anxiety, Affect, and Suicidality and Self-Harm Risk, from Individuals’ Own Descriptions

This thesis develops and evaluates language-based assessments that use artificial intelligence (AI) to transform open-ended language into quantitative indicators and descriptions of mental health related constructs. While closed-ended scales have long dominated psychological assessment, they are limited by fixed response formats and may not fully capture the complexity of individuals’ experiences. By contrast, language offers a flexible, and expressive medium for describing thoughts, emotions, and behaviors. Across four papers, this thesis examines whether language-based assessments can provide valid, and reliable tools for assessing psychological constructs such as depression, anxiety and affect, as well as mental health related risk assessments including suicidality and self-harm.Paper I compares four different language response formats—from selecting predefined words to producing full-text responses. We evaluated the response formats in terms of their validity—covering concurrent, incremental, face, discriminant, and external aspects—and their reliability, including test-retest and performance in a prospective sample. Using the Sequential Evaluation with Model Pre-registration (SEMP) approach, machine learning models were trained on a development dataset (N = 963) and pre- registered before being tested on a separate prospective sample (N = 145). These pre-registered models demonstrated moderate to strong validity and reliability, achieving predictive accuracy in the new sample (r = .60–.79). The consistent performance across formats suggests that they may be selected based on specific research or potential practical requirements.Paper II evaluates AI-based language models to evaluate the risk of suicide and self-harm based on individuals’ open-ended narratives about suicidality, self-harm, depression, anxiety, and overall mental health. Employing the SEMP framework, models were trained (N = 641) and pre-registered, then validated in a held-out set (N = 150) against expert ratings generated using the Longitudinal Expert Data (LED) approach. In a held-out test set, the language-based assessments showed alignment with expert ratings for suicidality (r = .70) and self-harm (r = .68), and significantly outperformed models that relied on demographic data.Paper III evaluates the causal validity of language-based assessments in an experimental setting. Few studies have tested whether language-based assessments can detect causal changes. In this randomized mixed-design experiment (N = 892), participants underwent mood induction in physical settings (N = 153) or via online videos (N = 739) across three conditions (church, mall, park). They reported pre- and post-mood affect using both open-ended responses and closed-ended Positive and Negative Affect (PANAS) ratings. We compared how well PANAS and language-based assessments classified the conditions. Language-based assessments outperformed PANAS in predictive accuracy across training (AUC = .74 vs. .63), online (AUC = .76 vs. .70), and offline holdout samples (AUC = .67 vs. .53). In addition, language-based assessments provided qualitative insights by visualizing word-level patterns across conditions.Paper IV introduces the L-BAM Library—an open repository for pre-validated language-based assessment models—and outlines a framework for sharing and applying these tools in transparent and reproducible ways. This paper emphasizes responsible open- science practices and encourages the independent validation of LBAs in new populations and contexts.In sum, this thesis demonstrates that language-based assessments can serve as valid, reliable, and informative research tools for measuring and describing mental health-related constructs. By leveraging the expressiveness of open-ended language, these methods address limitations of traditional scales and offer new possibilities for capturing complex psychological phenomena. Through systematic validation across diverse samples and contexts—including expert-rated risk assessment, experimental manipulations, and real-world implementation—this work contributes to the methodological advances of language-based methods into the broader landscape of psychological assessment and highlights the importance of transparent, cumulative practices for their continued development.