Abstract 4909: Artificial intelligence (AI) chatbots and their reponses to most searched Spanish cancer questions

Abstract Background: AI chatbots are predominantly trained on English contents. They perform well in answering English cancer questions, but their performance in other languages (such as Spanish) is unknown. Spanish-speaking patients are also concerned that they must use the paywall versions to get better responses, which may exacerbate existing cancer disparities. Methods: We evaluated the responses of AI chatbots to most searched Spanish cancer questions. Using Google Trends (1/1/2020-1/1/2024), we identified the top 5 most searched Spanish cancer questions related to the top 3 common cancers in US Hispanics/Latinos. We selected 6 popular AI chatbots (free and paywall versions of ChatGPT, Claude, and Gemini) and then generated 90 Spanish responses. Board-certified oncologists speaking native Spanish assessed the quality using DISCERN Instrument (score from 1 [low quality] to 5 [high quality]), actionability using Patient Education Materials Assessment Tool (score from 0 [no clear action suggestions] to 100% [clear action suggestions]), readability using Fernández Huerta Reading Grade Level (score from 1 [1st grade] to 13 [college]). Results: The quality of overall AI chatbot responses was moderate (mean [95% CI]: 3.5 [3.4-3.6]). The actionability was low (mean [95% CI]: 35.6% [30.8%-40.3%]), and the readability was high-school level (mean [95% CI]: 9.2 [8.8-9.6] grade). The performance of quality, actionability, and readability did not differ by free and paywall versions (P >0.05). Conclusions: AI chatbots provided moderately accurate information for most searched Spanish cancer-related questions. The responses were not readily actionable and written at the high-school level, which was not concordant with the American Medical Association’s recommendation (6th grade or lower). The performance did not improve by using the paywall versions. Relevance: To reduce cancer disparities in health literacy, AI chatbots need improvement in responding to Spanish cancer questions. Citation Format: En Cheng, Jesus D. Anampa, Carolina Bernabe-Ramirez, Juan Lin, Xiaonan Xue, Alyson B. Moadel-Robblee, Edward Chu. Artificial intelligence (AI) chatbots and their reponses to most searched Spanish cancer questions [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 4909.