Evaluating large language model responses to public questions on perianal abscess and anal fistula: a chart-informed comparative study

Large language models (LLMs) are increasingly used for health information, but their performance in anorectal conditions requiring urgent triage and nuanced surgical counseling remains uncertain. We evaluated LLM responses to public questions about perianal abscess and anal fistula, focusing on clinically consequential safety pathways and citation integrity. A purposively balanced 120-question benchmark was developed from Google Trends, de-identified hospital consultation and follow-up themes, public patient forums, and PubMed evidence searches. Six consumer-facing LLMs were queried once per question through retrieval-enabled web interfaces, yielding 720 responses. Three colorectal clinicians blinded to model identity rated safety, accuracy, clinician-rated patient-centered communication, DISCERN, Ensuring Quality Information for Patients (EQIP), Journal of the American Medical Association (JAMA) benchmark criteria, and the Global Quality Score (GQS); six formulas assessed readability. Paired comparisons used Cochran’s Q and Friedman tests. Potentially unsafe pairs underwent exploratory Grade 1–3 reclassification. A citation audit included 92 responses and 434 citations. Overall, 68/720 responses (9.4%) were potentially unsafe: 18 (2.5%) had a potential direct-harm pathway, 40 (5.6%) could delay or misdirect care, and 10 (1.4%) represented information limitations without an identified direct-harm or clinically meaningful delayed-care pathway. Grade-specific frequencies did not differ significantly among models. All models had a median accuracy score of 4.00; the overall difference was statistically significant but small (Kendall’s W = 0.128). Clinician-rated patient-centered communication varied more across models ( W = 0.444). Of 434 audited citations, 110 (25.3%) were authentic and bibliographically accurate, 205 (47.2%) were identifiable but materially inaccurate, 86 (19.8%) were not found or likely fabricated as stated, and 33 (7.6%) lacked sufficient identifying information. Qwen generated no citation-containing responses. Readability remained demanding overall, with Kimi showing the greatest formula-estimated burden and Qwen showing favorable values on several indices. Models showed broadly similar accuracy but shared clinically important limitations in escalation guidance, special-population counseling, evidence transparency, and citation reliability; no model was uniformly superior. LLMs may supplement but should not replace clinical evaluation. Patient-facing systems should prioritize explicit escalation guidance, verifiable sources, balanced treatment explanations, and patient-validated plain language.