S2959 Time for a Gut Check: Publication Bias in GI-Focused LLM Research

Introduction: There is growing interest in the use of artificial intelligence in clinical medicine, with a particular focus on large language models (LLMs) and their utility in patient communication and clinical decision-making. In gastroenterology (GI), numerous studies have evaluated LLMs, reflecting excitement around the technology. However, this enthusiasm raises concerns about potential bias toward positive findings. We conducted a systematic review to assess the presence of publication bias in experimental studies evaluating LLM applications in GI. Methods: We performed a systematic review using PubMed, Embase, and Cochrane Reviews to identify studies on LLMs published in the top 50 GI journals ranked by h-index between 2022 (the release year of ChatGPT) and April 2025. Each paper was evaluated as experimental or not, (i.e. testing LLMs’ ability to respond to questions vs discussing the implications of LLMs in GI). For experimental studies, we classified the study by experiment type and identified the most frequent experimental designs. Funnel plot analysis and Egger’s regression test were used to assess publication bias for the most common experiment type. Results: We identified 92 eligible papers, 43 of which were experimental. The most common experiment types were: (1) answering representative patient questions (n = 16), (2) assessing LLMs’ understanding of clinical guidelines (n = 7), and (3) evaluating LLMs’ ability to support clinical decision (n = 7). OpenAI’s ChatGPT was the most frequently evaluated LLM. We further analyzed studies assessing LLMs responses to representative patient questions. Funnel plot analysis showed asymmetry, with a lack of small studies reporting lower effect sizes, consistent with presence of publication bias. Egger’s test yielded a statistically significant result (P = 0.027), also indicating evidence of publication bias. Conclusion: Our analysis suggests the presence of publication bias in GI-LLM research, particularly in studies assessing patient-facing applications. The under-reporting of negative findings could skew clinical perceptions of LLM efficacy and safety, potentially leading to premature or overconfident clinical integration. Greater transparency and publication of null or negative results are needed to ensure a balanced evidence base. Limitations of this study include the relatively small sample size of GI-specific LLM studies and the focus on high-impact journals, which may further amplify bias.