REG2024: a benchmark challenge and curated dataset for pathology report generation from giga- pixel whole-slide images of bladder cancer
Background We aimed to construct a curated whole-slide image (WSI)-report dataset for bladder cancer and establish a clinically grounded benchmark for evaluating vision–language models through an open community challenge. Methods We curated a dataset of 902 deidentified bladder cancer WSIs paired with corresponding pathology reports. Seven teams submitted distinct modeling frameworks. Performance was assessed using a clinically weighted composite metric integrating word-overlap scores (ROUGE-L, BLEU-4), keyword matching, and embedding-based semantic similarity. Expert mean opinion scores (MOS) were obtained, and the correlation between the weighted-sum score and expert-assigned MOS was analyzed. Results Two modeling strategies emerged: end-to-end generative architectures and multi-step classification-then-generation pipelines. Among individual submissions, the top-ranked multi-step pipeline achieved the highest main-diagnosis accuracy (48/50), whereas end-to-end models showed greater sensitivity to rare pathological features. The weighted-sum metric correlated strongly with MOS within each team (Spearman’s ρ, 0.65–0.91) and, on reports with clinically important misclassifications, agreed with MOS more closely than any single metric. Conclusions Performance variability was influenced by architectural design and the integration of pretrained foundation models. Our proposed evaluation framework addressed key limitations of conventional natural language processing metrics in evaluating pathology reports. This challenge establishes a benchmark for clinical-grade pathology report generation by providing vision-language pathology datasets and introducing a clinically aligned evaluation framework, thereby advancing computational pathology.
Authors
- Harim Oh (ORCID: https://orcid.org/0000-0003-4904-4015)
- Hyo Sup Shim (ORCID: https://orcid.org/0000-0002-5718-3624)
- Kyung A. Kim (ORCID: https://orcid.org/0009-0004-9466-113X)
- Won‐Ki Jeong (ORCID: https://orcid.org/0000-0002-9393-6451)
- Thanaporn Viriyasaranon (ORCID: https://orcid.org/0000-0003-1408-9651)
- Hyeseong Lee (ORCID: https://orcid.org/0000-0002-7137-8874)
- Keonwoo Noh
- Wonkyeong Lee (ORCID: https://orcid.org/0009-0007-1758-9738)
- Joonbeom Park (ORCID: https://orcid.org/0000-0003-2907-0785)
- Jong-Ho Kim (ORCID: https://orcid.org/0000-0002-1309-0821)
- Jang-Hwan Choi (ORCID: https://orcid.org/0000-0001-5808-5252)
- Eunbeen Jo (ORCID: https://orcid.org/0000-0002-2468-0114)
- Sung Hak Lee
- Sehwan Yoo (ORCID: https://orcid.org/0009-0004-8951-2041)
- YuMi Lee
- Eunsu Kim
- Sungman Hong
- Sangjeong Ahn
- Yeongjae Shin
- Kyeongman Park
- Yousun Kang
- Hyun Yee Cho
- Daehong Lee
Institutions
- Ewha Womans University (KR)
- Seoul National University (KR)
- Nanyang Technological University (SG)
- Yonsei University (KR)
- Korea University (KR)
- Tokyo Polytechnic University (JP)
- Rivis (United States) (US)
- Korea University Guro Hospital (KR)
- Korea University Anam Hospital (KR)
- Catholic University of Korea (KR)
- Stanford University (US)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-18
- DOI
- https://doi.org/10.21203/rs.3.rs-10824156/v1
- Primary Topic
- AI in cancer detection
- Type
- preprint
Funders
- National Research Foundation
- National Research Foundation of Korea
- Ministry of Science and ICT, South Korea
- Korea University Anam Hospital