AI-driven versus human evaluations of psychotherapeutic communication performance: The impact of procedural prompting on assessment reliability and validity

This study examines whether large language models (LLMs) can reliably assess psychotherapeutic communication competencies. Specifically, it compares assessments of psychotherapeutic professional with two LLM-based rating approaches, a segment-based utterance-by-utterance evaluation of psychotherapeutic interactions and a holistic assessment of the conversations as a whole. Drawing on performance recordings of 37 psychology students from an examination involving two interaction scenarions with simulation patients, the investigation compared reliability and validity of assessments of empathy and communication structure among the LLM-based (using ChatGPT 1o) and human assessments. Results revealed that the segment-based approach attained reliability and validity levels comparable to those of trained psychotherapeutic assessors, particularly in empathy ratings, whereas the holistic approach consistently underperformed. These findings underscore the promise of integrating LLM-driven tools into psychotherapeutic training and performance assessments, suggesting that carefully calibrated prompt engineering may serve as a cost-effective adjunct to human judgment in high-stakes educational contexts. • LMMs can reliably and validly assess psychotherapy communication performance. • LLMs can match expert assessments of psychotherapy communication. • Evaluating communications utterance-by-utterance enhance assessment quality. • LMM based assessments can reduce instructor workload in educational settings.