Assessing generalizability of an AI-based visual test for cervical cancer screening

A number of challenges hinder artificial intelligence (AI) models from effective clinical translation. Foremost among these challenges is the lack of generalizability, which is defined as the ability of a model to perform well on datasets that have different characteristics from the training data. We recently investigated the development of an AI pipeline on digital images of the cervix, utilizing a multi-heterogeneous dataset of 9,462 women (17,013 images) and a multi-stage model selection and optimization approach, to generate a diagnostic classifier able to classify images of the cervix into "normal", "indeterminate" and "precancer/cancer" (denoted as "precancer+") categories. In this work, we investigate the performance of this multiclass classifier on external data not utilized in training and internal validation, to assess the generalizability of the classifier when moving to new settings. We assessed both the classification performance and repeatability of our classifier model across the two axes of heterogeneity present in our dataset: image capture device and geography, utilizing both out-of-the-box inference and retraining with external data. Our results demonstrate that device-level heterogeneity affects our model performance more than geography-level heterogeneity. Classification performance of our model is strong on images from a new geography without retraining, while incremental retraining with inclusion of images from a new device progressively improves classification performance on that device up to a point of saturation. Repeatability of our model is relatively unaffected by data heterogeneity and remains strong throughout. Our work supports the need for optimized retraining approaches that address data heterogeneity (e.g., when moving to a new device) to facilitate effective use of AI models in new settings.

Authors

Institutions

Publication Details

Journal
PLOS Digital Health
Published
2024-10-02
DOI
https://doi.org/10.1371/journal.pdig.0000364
Citations
10
Primary Topic
AI in cancer detection
Type
article
Field-Weighted Citation Impact
2.78

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Assessing generalizability of an AI-based visual test for cervical cancer screening

Jayashree Kalpathy-Cramer, Sílvia de Sanjosé, Karla Alfaro, Kanan Desai et al.
10 citations
PLOS Digital Health
AI in cancer detection
2.78
article

Assessing generalizability of an AI-based visual test for cervical cancer screening

Jayashree Kalpathy-Cramer, Sílvia de Sanjosé, Karla Alfaro, Kanan Desai, Syed Rakin Ahmed, Ana Cecilia Rodríguez, Joël Fokom Domgue, José Jerónimo, Didem Egemen, Rebecca Luckett, Brian Befano, Rakiya Saidu, Carolina Terán, Kittipat Charoenkwan, Tainá Raiol, Julia C. Gage, Chemtai Mungo, Ana Ribeiro, Mark Schiffman
article en
10 citations

Abstract

A number of challenges hinder artificial intelligence (AI) models from effective clinical translation. Foremost among these challenges is the lack of generalizability, which is defined as the ability of a model to perform well on datasets that have different characteristics from the training data. We recently investigated the development of an AI pipeline on digital images of the cervix, utilizing a multi-heterogeneous dataset of 9,462 women (17,013 images) and a multi-stage model selection and optimization approach, to generate a diagnostic classifier able to classify images of the cervix into "normal", "indeterminate" and "precancer/cancer" (denoted as "precancer+") categories. In this work, we investigate the performance of this multiclass classifier on external data not utilized in training and internal validation, to assess the generalizability of the classifier when moving to new settings. We assessed both the classification performance and repeatability of our classifier model across the two axes of heterogeneity present in our dataset: image capture device and geography, utilizing both out-of-the-box inference and retraining with external data. Our results demonstrate that device-level heterogeneity affects our model performance more than geography-level heterogeneity. Classification performance of our model is strong on images from a new geography without retraining, while incremental retraining with inclusion of images from a new device progressively improves classification performance on that device up to a point of saturation. Repeatability of our model is relatively unaffected by data heterogeneity and remains strong throughout. Our work supports the need for optimized retraining approaches that address data heterogeneity (e.g., when moving to a new device) to facilitate effective use of AI models in new settings.

PLOS Digital HealthVol. 3(10)
Dartmouth College (US), South African Medical Research Council (ZA), University of North Carolina at Chapel Hill (US), National Institutes of Health (US), Beth Israel Deaconess Medical Center (US), Université de Yaoundé I (CM), The University of Texas MD Anderson Cancer Center (US), Harvard University (US), University of Washington (US), Harvard University Press (US), Office of the Director (US), Information Management Services (US), Cameroon Baptist Convention Health Services (CM), University of Saint Francis Xavier (BO), Athinoula A. Martinos Center for Biomedical Imaging (US), National Cancer Institute (US), Barcelona Institute for Global Health (ES), Department of Health and Human Services (US), Chiang Mai University (TH), University of Colorado Anschutz Medical Campus (US), Fundação Oswaldo Cruz (BR), Massachusetts Institute of Technology (US)
National Institutes of Health, National Cancer Institute, Moonshot Research and Development Program, Cancer Moonshot
Reduced inequalities
Openalex Percentile: Top 8%
AI in cancer detection
2.78
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.