Theory-Grounded Evaluation of Multi-Turn Large Language Model Communication for Cancer Screening: Framework Development and Evaluation Study (Preprint)

BACKGROUND Counseling a person about cancer screening requires more than factually accurate information. It must respond to the person's readiness to act, beliefs, barriers, emotions, and sociocultural context while conveying guideline-concordant recommendations. Large language model (LLM)–based health communication is often evaluated with general measures of fluency, coherence, or isolated factual accuracy, which may not show whether a multi-turn conversation is clinically accurate, complete, or patient centered. OBJECTIVE This study aimed to develop a theory-grounded framework for structuring and evaluating multi-turn LLM-based breast and cervical cancer screening communication and to apply it to identify where conversational quality diverges from clinically relevant and patient-centered quality. METHODS Constructs from the Transtheoretical Model and the Health Belief Model, intrapersonal and interpersonal (sociocultural) determinants of screening, patient-centered communication functions, and age-specific screening guidance were operationalized in a structured knowledge representation with patient and provider branches. Role-specific GPT-4o Patient and Provider Agents instantiated this representation as a controlled simulation testbed, with stage-aware provider communication, dialogue state tracking, and retrieval of guideline text, and generated 2000 synthetic patient-provider dialogues. The synthetic conversations were not assumed or validated to represent real patient-provider conversations. Dialogues were evaluated with Fine-grained Evaluation of Dialog (FED), rubric-based G-Eval, and human review of 100 selected dialogues by two annotators on six accuracy, completeness, and communication criteria, each scored from −1 to 1. RESULTS FED scores were highest for coherence (0.98), error recovery (0.97), and consistency (0.94) and lowest for likeability (0.45) and informativeness (0.47). G-Eval scores ranged from 2.34 to 2.50 on a 1-3 scale. A total of 83 dialogues had complete paired human ratings, with a pooled composite score of 0.59 (95% CI 0.56-0.63). Among the human-rated criteria, patient-centered concern had the highest pooled mean (0.87) and cancer screening accuracy the lowest (0.28), followed by response completeness (0.46). Overall interannotator agreement was fair (linear-weighted Cohen κ=0.29); criterion-level agreement was highest for cancer screening accuracy (κ=0.52) and near zero for emotional acknowledgment (κ=0.08) and patient-centered concern (κ=−0.05). CONCLUSIONS Behavioral and communication theory can be operationalized into an interpretable structure for examining multi-turn LLM cancer screening communication. In this evaluation, high automated scores for conversational continuity coexisted with low human ratings of cancer screening accuracy and response completeness. Conversational fluency and coherence should therefore not be interpreted as evidence of clinically accurate or patient-centered health communication. The findings come from one model configuration, synthetic dialogues, and two annotators, without a baseline or ablation; validation against real conversations, by clinicians and community members, and against patient outcomes is needed before any patient-facing use.

Authors

Publication Details

Published
2026-11-20
DOI
https://doi.org/10.2196/preprints.117464
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Theory-Grounded Evaluation of Multi-Turn Large Language Model Communication for Cancer Screening: Framework Development and Evaluation Study (Preprint)

Victoria Churchill, Muhammed Yassin Idris, Ugur Kursuncu, Agnik Saha et al.
Artificial Intelligence in Healthcare and Education
preprint

Theory-Grounded Evaluation of Multi-Turn Large Language Model Communication for Cancer Screening: Framework Development and Evaluation Study (Preprint)

Victoria Churchill, Muhammed Yassin Idris, Ugur Kursuncu, Agnik Saha, Alexander Eaves
preprint en

Abstract

BACKGROUND Counseling a person about cancer screening requires more than factually accurate information. It must respond to the person's readiness to act, beliefs, barriers, emotions, and sociocultural context while conveying guideline-concordant recommendations. Large language model (LLM)–based health communication is often evaluated with general measures of fluency, coherence, or isolated factual accuracy, which may not show whether a multi-turn conversation is clinically accurate, complete, or patient centered. OBJECTIVE This study aimed to develop a theory-grounded framework for structuring and evaluating multi-turn LLM-based breast and cervical cancer screening communication and to apply it to identify where conversational quality diverges from clinically relevant and patient-centered quality. METHODS Constructs from the Transtheoretical Model and the Health Belief Model, intrapersonal and interpersonal (sociocultural) determinants of screening, patient-centered communication functions, and age-specific screening guidance were operationalized in a structured knowledge representation with patient and provider branches. Role-specific GPT-4o Patient and Provider Agents instantiated this representation as a controlled simulation testbed, with stage-aware provider communication, dialogue state tracking, and retrieval of guideline text, and generated 2000 synthetic patient-provider dialogues. The synthetic conversations were not assumed or validated to represent real patient-provider conversations. Dialogues were evaluated with Fine-grained Evaluation of Dialog (FED), rubric-based G-Eval, and human review of 100 selected dialogues by two annotators on six accuracy, completeness, and communication criteria, each scored from −1 to 1. RESULTS FED scores were highest for coherence (0.98), error recovery (0.97), and consistency (0.94) and lowest for likeability (0.45) and informativeness (0.47). G-Eval scores ranged from 2.34 to 2.50 on a 1-3 scale. A total of 83 dialogues had complete paired human ratings, with a pooled composite score of 0.59 (95% CI 0.56-0.63). Among the human-rated criteria, patient-centered concern had the highest pooled mean (0.87) and cancer screening accuracy the lowest (0.28), followed by response completeness (0.46). Overall interannotator agreement was fair (linear-weighted Cohen κ=0.29); criterion-level agreement was highest for cancer screening accuracy (κ=0.52) and near zero for emotional acknowledgment (κ=0.08) and patient-centered concern (κ=−0.05). CONCLUSIONS Behavioral and communication theory can be operationalized into an interpretable structure for examining multi-turn LLM cancer screening communication. In this evaluation, high automated scores for conversational continuity coexisted with low human ratings of cancer screening accuracy and response completeness. Conversational fluency and coherence should therefore not be interpreted as evidence of clinically accurate or patient-centered health communication. The findings come from one model configuration, synthetic dialogues, and two annotators, without a baseline or ablation; validation against real conversations, by clinicians and community members, and against patient outcomes is needed before any patient-facing use.

Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon