Abstract B163: Evaluation of large language models for the translational equivalency of oncology informed consent forms from English to Spanish

Abstract Approximately 8% of the US population speaks a language other than English as their primary language. Limited English Proficiency (LEP) contributes to the underrepresentation of Hispanic participants in cancer clinical trials. Although professional translation services exist, they remain difficult to access, time consuming and cost prohibitive, particularly for investigator-initiated trials. Machine-generated translations of Informed Consent Forms (ICFs) could provide a low-cost approach to expanding access to clinical trials. But there is limited data on the accuracy and safety of Large Language Models (LLMs) in translation of ICFs. Therefore, we sought to evaluate the language equivalence of English to Spanish translation of cancer trial ICFs created using LLMs. We tested 2 general-use LLMs, DeepL Pro and ChatGPT-4o, and a medically trained LLM, Med English-2-Spanish, on the translation of 3 oncology clinical trial ICFs. LLM-translated ICFs were compared to certified professional translations. To evaluate translational accuracy and impact on patient safety, we developed a 5-point Likert Scale on 5 domains: Semantic, Idiomatic, Experiential, Conceptual, and Safety. A score of 5 represented complete language equivalence between ICFs, whereas 1 represented a lack of language equivalence. Language equivalence is an established measure to ensure the translated document has the same syntactical, conceptual, and grammatical meaning in the target culture. Two native Spanish-speaking bilingual board-certified oncologists independently scored translated ICFs. Weighted Cohen’s Kappa statistic was calculated to determine interrater reliability. We compared differences in LLM translations versus the professional translation via a two Sample T-Test. This study was deemed exempt by the University of Pennsylvania’s Institutional Review Board. Weighted Cohen’s Kappa (0.95, 0.85 – 0.97) exhibited a high degree of agreement on scoring between the two evaluators. Firm translations exhibited the highest overall accuracy (mean = 4.99, SD = 0.02) on all equivalences. DeepL Pro scored relatively well (mean = 4.43, SD = 0.07); however, accuracy was significantly lower than certified translation (p < 0.001). GPT4o demonstrated a high degree of accuracy (mean = 4.89, SD = 0.17). Med English-to-Spanish demonstrated lowest accuracy (mean = 3.32, SD = 0.40) and difference in the means compared to the certified translation which was statistically significant (p < 0.001). Low-cost LLM translations of ICFs are fairly accurate using DeepL Pro and ChatGPT-4o and do not appear to compromise patients’ comprehension of trial interventions and ability to enroll. GPT-4o outperformed Med English-2-Spanish moderately and DeepL slightly. The results of this exploratory study suggest GPT-4o’s high degree of accuracy merits further study as a possible strategy for reducing financial and time burden of cancer ICF translation, thereby, potentially reducing barriers to LEP population enrollment in cancer clinical trials. Citation Format: Shaurya Khanna, Sandra P . Susanibar-Adaniya, Ximena Jordan Bruno, Gary Weissman, Carmen E. Guerra. Evaluation of large language models for the translational equivalency of oncology informed consent forms from English to Spanish [abstract]. In: Proceedings of the 18th AACR Conference on the Science of Cancer Health Disparities; 2025 Sep 18-21; Baltimore, MD. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2025;34(9 Suppl):Abstract nr B163.