Run-to-run variability of large language models extracting structured data from Turkish colorectal cancer pathology report summaries
Large language models are increasingly used to convert free-text pathology reports into structured data, but validation usually relies on a single query. We submitted an identical zero-shot prompt containing 75 semi-structured, single-centre Turkish colorectal cancer pathology report summaries to ChatGPT, Gemini and DeepSeek through public web interfaces, ten times each in new conversations at two time points, and compared eight extracted variables with a manually curated reference standard. Accuracy was at ceiling: 14,972 of 15,000 fields (99.8%) were correct across direct-extraction runs, and a regular-expression program written by Gemini was equally accurate, showing that the task is solvable by pattern matching and cannot discriminate between models on comprehension. Repeated querying nevertheless revealed variability invisible in single runs. In one of ten runs, ChatGPT left tumour differentiation empty in 28 records; within that run, omission was strongly associated with source-text spelling (28/29 versus 0/26 records). In five of ten runs, Gemini generated and executed parsing code, with output indistinguishable from direct extraction. Output layout varied between runs and over time. These findings establish that such instability occurs, not its frequency. Single-query accuracy does not establish reproducibility. For structured data extraction, outputs from public chat services should not be used without verification.
Authors
- Ethem Bilgiç (ORCID: https://orcid.org/0000-0002-3144-868X)
- Erkan Karacan (ORCID: https://orcid.org/0000-0001-8081-5944)
Institutions
- Karabük University (TR)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-13
- DOI
- https://doi.org/10.1038/s41598-026-77778-x
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00