Towards clinical communication training with LLM-based conversational virtual patients and automatic evaluation systems

Clinical communication is a fundamental skill that impacts patient satisfaction, treatment adherence, and clinical outcomes. Traditional training methods, such as standardized patients (SPs) or virtual standardized patients (VSPs), face barriers including high costs, logistical complexity, and lack of spontaneous interaction. This study addresses these challenges by developing a scalable alternative based on Large Language Models (LLMs) to train medical students in the anamnesis phase. We developed an interactive Conversational Virtual Patient (CVP) system, understood here as a text- and voice-based simulated patient without graphical embodiment. The conversational logic is orchestrated as an explicit state machine over the interview, implemented with LangGraph, a graph-based framework in which each node performs one processing stage and a shared state persists across turns. Clinical consistency across the interview is maintained by a semantic memory layer implemented with LangMem, a library that stores clinical facts as vector embeddings and retrieves them by similarity. Performance is measured by an automatic evaluation module assessing 24 communication criteria derived from validated instruments such as the Communication Assessment Tool (CAT) and the Communication Skills Scale (CSS). The platform was validated with 21 participants, comprising 9 eighth-semester medical students, 11 practising physicians, and 1 expert in healthcare technology. All sessions were conducted in Spanish. Considering the student cohort, which addresses the educational utility of the tool, the system obtained a Chatbot Usability Questionnaire (CUQ) score of 82.3/100 and an average acceptance of the automated feedback of 4.50/5. The physician and expert cohort, which provides expert clinical validation, obtained 96.5/100 and 4.21/5 respectively, yielding a pooled usability score of 90.4/100 and a pooled feedback acceptance of 4.33/5. In an exploratory comparison against eleven physician ratings of the same interviews, rank agreement was highest for “Interest” ( $$\rho = 0.64$$ ) and lowest for “Completeness” ( $$\rho = 0.09$$ ), and the automated scores fell below the physician scores in every dimension. No dimension attained significance after correction for six comparisons, and the confidence intervals were wide, so these coefficients describe the observed sample rather than the agreement attainable in general. The results support the feasibility of the system as a complement to traditional medical training, providing immediate and standardized feedback. Given the sample size and the single-institution design, these findings should be read as an initial validation of feasibility and acceptability rather than as evidence of pedagogical efficacy.