Large language model use and hallucination awareness among medical students: a single-centre cross-sectional survey

Abstract Background Large language models (LLMs) are increasingly used in medical learning, but their answers may contain plausible yet incorrect information. It remains unclear whether students’ awareness of these hallucinations is associated with how they judge the credibility of specialty-specific answers. Methods This single-centre cross-sectional survey was conducted at Sun Yat-sen University, China, from January to July 2026. Students completed a 17-item questionnaire on site and were classified into three groups by ophthalmic educational background. Students and five blinded senior ophthalmologists rated five examination-derived LLM answers on a five-point credibility scale. Analyses included descriptive statistics, Kendall’s concordance, Friedman and Kruskal–Wallis tests, and exploratory Spearman correlations with bootstrap confidence intervals and false-discovery-rate (FDR) correction. Results Of 145 invited students, 138 provided complete questionnaires (95.2%): 22 in the basic-knowledge group, 80 in the course-level group, and 36 in the specialist-level group. Overall, 127 (92.0%) had used an LLM and 121 (87.7%) perceived improved learning efficiency. Although 98 (71.0%) reported familiarity with hallucinations and 108 (78.3%) reported encountering them, only 35 (25.4%) felt able to identify them clearly. Student mean credibility ratings ranged from 3.51 to 3.67, compared with expert means of 3.0–5.0. Students rated the retinal-neovascularization answer 0.51 points above the expert mean and the traumatic-lens-dislocation answer 1.43 points below it. Hallucination familiarity was associated with LLM familiarity (ρ = 0.342), use frequency (ρ = 0.421), and self-rated detection ability (ρ = 0.274; all FDR-adjusted P < 0.01). Neither hallucination familiarity nor self-rated ability was significantly associated with closer agreement with experts. Conclusions Students commonly used LLMs and recognised hallucination risks, but few felt able to identify hallucinations clearly. Awareness was not significantly associated with closer agreement with expert credibility judgments. These findings support further evaluation of supervised, case-based teaching in source verification and appraisal of LLM-generated information. Objective hallucination detection requires separate assessment.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-11390972/v1
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Large language model use and hallucination awareness among medical students: a single-centre cross-sectional survey

Jiamin Ouyang, Zitian Liu, Yuxian Zou, Lingxuan Peng et al.
Research Square
Artificial Intelligence in Healthcare and Education
preprint

Large language model use and hallucination awareness among medical students: a single-centre cross-sectional survey

Jiamin Ouyang, Zitian Liu, Yuxian Zou, Lingxuan Peng, Le Gao
preprint en

Abstract

Abstract Background Large language models (LLMs) are increasingly used in medical learning, but their answers may contain plausible yet incorrect information. It remains unclear whether students’ awareness of these hallucinations is associated with how they judge the credibility of specialty-specific answers. Methods This single-centre cross-sectional survey was conducted at Sun Yat-sen University, China, from January to July 2026. Students completed a 17-item questionnaire on site and were classified into three groups by ophthalmic educational background. Students and five blinded senior ophthalmologists rated five examination-derived LLM answers on a five-point credibility scale. Analyses included descriptive statistics, Kendall’s concordance, Friedman and Kruskal–Wallis tests, and exploratory Spearman correlations with bootstrap confidence intervals and false-discovery-rate (FDR) correction. Results Of 145 invited students, 138 provided complete questionnaires (95.2%): 22 in the basic-knowledge group, 80 in the course-level group, and 36 in the specialist-level group. Overall, 127 (92.0%) had used an LLM and 121 (87.7%) perceived improved learning efficiency. Although 98 (71.0%) reported familiarity with hallucinations and 108 (78.3%) reported encountering them, only 35 (25.4%) felt able to identify them clearly. Student mean credibility ratings ranged from 3.51 to 3.67, compared with expert means of 3.0–5.0. Students rated the retinal-neovascularization answer 0.51 points above the expert mean and the traumatic-lens-dislocation answer 1.43 points below it. Hallucination familiarity was associated with LLM familiarity (ρ = 0.342), use frequency (ρ = 0.421), and self-rated detection ability (ρ = 0.274; all FDR-adjusted P < 0.01). Neither hallucination familiarity nor self-rated ability was significantly associated with closer agreement with experts. Conclusions Students commonly used LLMs and recognised hallucination risks, but few felt able to identify hallucinations clearly. Awareness was not significantly associated with closer agreement with expert credibility judgments. These findings support further evaluation of supervised, case-based teaching in source verification and appraisal of LLM-generated information. Objective hallucination detection requires separate assessment.

Research Square
Sun Yat-sen University (CN)
Sun Yat-sen University
Good health and well-being, Quality education
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon