Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional study

The capability of ChatGPT to understand and generate human-readable text has prompted the investigation of its potential as mental health assessment tools. This study aims to explore the validity of ChatGPT in assessing loneliness and online social support among college students by comparing scoring consistency between ChatGPT and the validated questionnaires. This was a cross-sectional study between June and August 2024. We pre-trained ChatGPT-4 based on the validated University of California Los Angeles Loneliness Scale-6 (ULS-6) and Chinese Youth Version of the Online Social Support Scale (OSSS-CS), creating a structured interview questionnaire. Participants were invited to complete both the ChatGPT-created questionnaire and the validated questionnaires. We used Spearman correlation analysis, Intra-class correlation coefficients (ICC), and Bland-Altman plots to assess the agreement between the scores from ChatGPT and the validated questionnaires. In addition, we evaluated ceiling and floor effects. A total of 216 college students participated the survey. The results demonstrated a good consistency between the scores obtained from ChatGPT and the validated questionnaires, with ICC of 0.81 (95% CI 0.75-0.85, p < 0.001) for ULS-6 and 0.95 (95% CI 0.94-0.96, p < 0.001) for OSSS-CS. The Spearman correlation coefficients were 0.64 (p < 0.001) for ULS-6 and 0.89 (p < 0.001) for OSSS-CS, indicating a moderate correlation. No ceiling or floor effects were observed. The ChatGPT-created questionnaire demonstrated acceptable consistency with the validated questionnaires. Future studies can further explore the performance of ChatGPT in different populations and domains, as well as how to integrate it with validated questionnaires to enhance the accessibility of assessments.