Cross-linguistic benchmarking of NLP metrics for psychosis research

Language is central to psychosis, making it a key focus for both clinical evaluation and research. Many automated linguistic metrics have been developed to analyze speech in psychosis, yet few studies have compared them systematically. In this study, we benchmarked the performance and robustness of 51 natural language processing (NLP) metrics using two datasets: one with German- and one with Russian-speaking participants. Metric performance was defined as the metrics' capacity to reflect symptom severity and distinguish individuals with psychosis from healthy controls. The best-performing metrics tended to be robust across different clinical rating scales and speech elicitation tasks. Overall, verbosity metrics (e.g., word count) outperformed most other metrics across tasks and languages. Thus, verbosity-based metrics should be used as a baseline when evaluating cross-linguistic NLP metrics that are both clinically meaningful and useful for psychosis research.