Impact of cancer outcome data source on the diagnostic accuracy of ovarian cancer prediction models: a primary care cohort study

Objectives Electronic health records are widely used to develop diagnostic prediction models for cancer. Some studies use cancer registry (CR) data, the gold standard for cancer case recordings, whereas others rely on data from alternative healthcare sources. We aimed to evaluate the impact of using CR and non-CR data sources on the diagnostic accuracy of the Ovatools ovarian cancer (OC) risk prediction model. Methods Retrospective cohort study using linked Clinical Practice Research Datalink (CPRD), hospital episodic statistics (HES) and CR data from women tested for cancer antigen 125 (CA125) in England (1 May 2011–31 December 2017). Ovatools model performance and diagnostic accuracy were compared when different data sources were used, alone and in combination, to identify the outcome, OC diagnosis in the year after CA125 testing. Threshold accuracy was measured at the National Institute for Health and Care Excellence ≥3% risk threshold. Results Among 340 769 CA125-tested women, OC incidence within 12 months was highest when using HES data (0.84%), compared with CR (0.75%) and CPRD (0.65%). Area under the curve was highest when using CR alone (0.924) and lower using CPRD (0.903) or CR+CPRD+HES (0.892). At a ≥3% risk threshold, sensitivity was highest when using CR data (73.2%) and lower using CPRD (68.8%). The positive predictive value was lowest using CPRD (13.8%) and highest using CPRD+CR+HES (19.4%). Conclusion Using an OC exemplar, we found moderate variation in model performance and threshold accuracy when different data sources were used to define cancer. To ensure cancer prediction models perform as expected in real world clinical practice, gold standard data sources, such as CR data, should be used for model development and validation.