Generative AI chatbots for reliable cancer information: Evaluating web-search, multilingual, and reference capabilities of emerging large language models
Recent advancements in large language models (LLMs) enable real-time web search, improved referencing, and multilingual support, yet ensuring they provide safe health information remains crucial. This perspective evaluates seven publicly accessible LLMs-ChatGPT, Co-Pilot, Gemini, MetaAI, Claude, Grok, Perplexity-on three simple cancer-related queries across eight languages (336 responses: English, French, Chinese, Thai, Hindi, Nepali, Vietnamese, and Arabic). None of the 42 English responses contained clinically meaningful hallucinations, whereas 7 of 294 non-English responses did. 48 % (162/336) of responses included valid references, but 39 % of the English references were.com links reflecting quality concerns. English responses frequently exceeded an eighth-grade level, and many non-English outputs were also complex. These findings reflect substantial progress over the past 2-years but reveal persistent gaps in multilingual accuracy, reliable reference inclusion, referral practices, and readability. Ongoing benchmarking is essential to ensure LLMs safely support global health information dichotomy and meet online information standards.
Authors
- Benjamin Chu (ORCID: https://orcid.org/0000-0003-2775-2336)
- Agnès Vitry (ORCID: https://orcid.org/0000-0003-1091-6453)
- Stephen Bacchi (ORCID: https://orcid.org/0000-0001-5130-8628)
- Ganessan Kichenadasse (ORCID: https://orcid.org/0000-0001-9923-5149)
- Adel Shahnam (ORCID: https://orcid.org/0000-0002-0043-8374)
- Andrew Rowland (ORCID: https://orcid.org/0000-0002-8946-3954)
- Lee X. Li (ORCID: https://orcid.org/0000-0001-5259-5198)
- Natansh D. Modi (ORCID: https://orcid.org/0000-0003-3262-5374)
- Warit Ruanglertboon (ORCID: https://orcid.org/0000-0003-0540-7427)
- Ashley M. Hopkins (ORCID: https://orcid.org/0000-0001-7652-4378)
- Michael J. Sorich (ORCID: https://orcid.org/0000-0003-1999-866X)
- Rakchha Chhetri (ORCID: https://orcid.org/0000-0003-2474-6547)
- Bradley D. Menz (ORCID: https://orcid.org/0000-0002-0855-5081)
- Ahmad Y. Abuhelwa (ORCID: https://orcid.org/0000-0002-4182-065X)
- Yuan Gao (ORCID: https://orcid.org/0000-0001-6647-2156)
Institutions
- Prince of Songkla University (TH)
- Flinders University (AU)
- University of South Australia (AU)
- Peter MacCallum Cancer Centre (AU)
- Flinders Medical Centre (AU)
- University of Sharjah (AE)
- Massachusetts General Hospital (US)
- Center for Systems Biology (US)
Publication Details
- Journal
- European Journal of Cancer
- Published
- 2025-02-04
- DOI
- https://doi.org/10.1016/j.ejca.2025.115274
- Citations
- 23
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 15.05
Funders
- Hospital Research Foundation
- Fulbright Australia
- Tour de Cure
- Medical Research Council
- National Health and Medical Research Council
- Cancer Council South Australia