Run-to-run variability of large language models extracting structured data from Turkish colorectal cancer pathology report summaries

Large language models are increasingly used to convert free-text pathology reports into structured data, but validation usually relies on a single query. We submitted an identical zero-shot prompt containing 75 semi-structured, single-centre Turkish colorectal cancer pathology report summaries to ChatGPT, Gemini and DeepSeek through public web interfaces, ten times each in new conversations at two time points, and compared eight extracted variables with a manually curated reference standard. Accuracy was at ceiling: 14,972 of 15,000 fields (99.8%) were correct across direct-extraction runs, and a regular-expression program written by Gemini was equally accurate, showing that the task is solvable by pattern matching and cannot discriminate between models on comprehension. Repeated querying nevertheless revealed variability invisible in single runs. In one of ten runs, ChatGPT left tumour differentiation empty in 28 records; within that run, omission was strongly associated with source-text spelling (28/29 versus 0/26 records). In five of ten runs, Gemini generated and executed parsing code, with output indistinguishable from direct extraction. Output layout varied between runs and over time. These findings establish that such instability occurs, not its frequency. Single-query accuracy does not establish reproducibility. For structured data extraction, outputs from public chat services should not be used without verification.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-13
DOI
https://doi.org/10.1038/s41598-026-77778-x
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Run-to-run variability of large language models extracting structured data from Turkish colorectal cancer pathology report summaries

Ethem Bilgiç, Erkan Karacan
2 links
Scientific Reports
Artificial Intelligence in Healthcare and Education
article

Run-to-run variability of large language models extracting structured data from Turkish colorectal cancer pathology report summaries

Ethem Bilgiç, Erkan Karacan
article en

Abstract

Large language models are increasingly used to convert free-text pathology reports into structured data, but validation usually relies on a single query. We submitted an identical zero-shot prompt containing 75 semi-structured, single-centre Turkish colorectal cancer pathology report summaries to ChatGPT, Gemini and DeepSeek through public web interfaces, ten times each in new conversations at two time points, and compared eight extracted variables with a manually curated reference standard. Accuracy was at ceiling: 14,972 of 15,000 fields (99.8%) were correct across direct-extraction runs, and a regular-expression program written by Gemini was equally accurate, showing that the task is solvable by pattern matching and cannot discriminate between models on comprehension. Repeated querying nevertheless revealed variability invisible in single runs. In one of ten runs, ChatGPT left tumour differentiation empty in 28 records; within that run, omission was strongly associated with source-text spelling (28/29 versus 0/26 records). In five of ten runs, Gemini generated and executed parsing code, with output indistinguishable from direct extraction. Output layout varied between runs and over time. These findings establish that such instability occurs, not its frequency. Single-query accuracy does not establish reproducibility. For structured data extraction, outputs from public chat services should not be used without verification.

Scientific Reports
Karabük University (TR)
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon