OC15.02: Development, validation and comparison of models predicting outcomes in pregnancies of unknown location using regression and machine learning models

To develop, validate and compare multiclass prediction models to predict ectopic pregnancy (EP) following a pregnancy of unknown location (PUL) classification. This was a secondary analysis of prospective and consecutive multicentre data. Dataset 1 includes 2894 patients recruited from eight early pregnancy assessment units in London between 2015 and 2017. Dataset 2 includes 1870 patients from two of the eight centres recruited between 2017 and 2021: St. Mary's (SMH) and Queen Charlotte's and Chelsea (QCCH). We developed models using three biomarkers and four clinical variables using six algorithms: standard multinomial logistic regression (MLR), Ridge MLR, random forest (RF), XGBoost, neural network (NN), and support vector machine (SVM). The models were developed and validated using internal-external cross validation (IECV) on dataset 1, and externally validated on data from both centres in dataset 2. In the development vs validation data, there were 12% vs 12% EP/persisting PUL (PPUL), 46% vs 37% failed PUL (FPUL), and 32% vs 37% intrauterine pregnancy (IUP), with 10% vs 14% PUL lost to follow-up. The area under the receiver operating characteristic (AUROC) curve for EP/PPUL vs other ranged from 0.87-0.89 on IECV, from 0.81-0.84 on QCCH, and from 0.87-0.89 on SMH. The AUROC for IUP vs FPUL ranged from 0.97 to 0.99 for all models on all validations. Calibration was roughly similar for most models. Net benefit at 5% risk threshold of EP/PPUL was similar as well, although Ridge MLR had the lowest value on all validations. For 14 to 15% of patients, the estimated risk of EP/PPUL varied 20 percentage points or more depending on choice of algorithm. Most models performed similarly well, however the estimated risks for an individual varied between models.