Theory-Grounded Evaluation of Multi-Turn Large Language Model Communication for Cancer Screening: Framework Development and Evaluation Study (Preprint)
BACKGROUND Counseling a person about cancer screening requires more than factually accurate information. It must respond to the person's readiness to act, beliefs, barriers, emotions, and sociocultural context while conveying guideline-concordant recommendations. Large language model (LLM)–based health communication is often evaluated with general measures of fluency, coherence, or isolated factual accuracy, which may not show whether a multi-turn conversation is clinically accurate, complete, or patient centered. OBJECTIVE This study aimed to develop a theory-grounded framework for structuring and evaluating multi-turn LLM-based breast and cervical cancer screening communication and to apply it to identify where conversational quality diverges from clinically relevant and patient-centered quality. METHODS Constructs from the Transtheoretical Model and the Health Belief Model, intrapersonal and interpersonal (sociocultural) determinants of screening, patient-centered communication functions, and age-specific screening guidance were operationalized in a structured knowledge representation with patient and provider branches. Role-specific GPT-4o Patient and Provider Agents instantiated this representation as a controlled simulation testbed, with stage-aware provider communication, dialogue state tracking, and retrieval of guideline text, and generated 2000 synthetic patient-provider dialogues. The synthetic conversations were not assumed or validated to represent real patient-provider conversations. Dialogues were evaluated with Fine-grained Evaluation of Dialog (FED), rubric-based G-Eval, and human review of 100 selected dialogues by two annotators on six accuracy, completeness, and communication criteria, each scored from −1 to 1. RESULTS FED scores were highest for coherence (0.98), error recovery (0.97), and consistency (0.94) and lowest for likeability (0.45) and informativeness (0.47). G-Eval scores ranged from 2.34 to 2.50 on a 1-3 scale. A total of 83 dialogues had complete paired human ratings, with a pooled composite score of 0.59 (95% CI 0.56-0.63). Among the human-rated criteria, patient-centered concern had the highest pooled mean (0.87) and cancer screening accuracy the lowest (0.28), followed by response completeness (0.46). Overall interannotator agreement was fair (linear-weighted Cohen κ=0.29); criterion-level agreement was highest for cancer screening accuracy (κ=0.52) and near zero for emotional acknowledgment (κ=0.08) and patient-centered concern (κ=−0.05). CONCLUSIONS Behavioral and communication theory can be operationalized into an interpretable structure for examining multi-turn LLM cancer screening communication. In this evaluation, high automated scores for conversational continuity coexisted with low human ratings of cancer screening accuracy and response completeness. Conversational fluency and coherence should therefore not be interpreted as evidence of clinically accurate or patient-centered health communication. The findings come from one model configuration, synthetic dialogues, and two annotators, without a baseline or ablation; validation against real conversations, by clinicians and community members, and against patient outcomes is needed before any patient-facing use.
Authors
- Victoria Churchill (ORCID: https://orcid.org/0000-0003-0393-3212)
- Muhammed Yassin Idris (ORCID: https://orcid.org/0000-0002-3899-3728)
- Ugur Kursuncu (ORCID: https://orcid.org/0000-0002-8108-9590)
- Agnik Saha (ORCID: https://orcid.org/0009-0002-5102-8559)
- Alexander Eaves
Publication Details
- Published
- 2026-11-20
- DOI
- https://doi.org/10.2196/preprints.117464
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- preprint