CEILING EFFECT OF THE COMBINED NORWEGIAN AND DANISH KNEE LIGAMENT REGISTERS LIMITS ANTERIOR CRUCIATE LIGAMENT RECONSTRUCTION OUTCOME PREDICTION

Clinical tools based on machine learning analysis now exist for outcome prediction following primary anterior cruciate ligament reconstruction (ACLR). Relying partly on data volume, a general principle is that more data may lead to improved model accuracy. The purpose of this study was to apply machine learning to a combined dataset comprised of the Norwegian and Danish knee ligament registers (KLR) with the aim of producing an algorithm that can predict subsequent revision surgery with improved accuracy relative to a previously published model developed using only the Norwegian register. The hypothesis was that the additional patient data would result in an algorithm that is more accurate. Machine learning analysis was performed on the combined KLR. The primary outcome was the probability of revision ACL reconstruction within 1, 2, and 5 years. Data was split randomly into training sets (75%) and test sets (25%). Four machine learning models intended for this type of data were tested: Cox Lasso, survival random forest, gradient boosted regression (GBM), and super learner. Model performance was evaluated by calculating concordance and calibration using methods adapted for censored data. Concordance measures the proportion of pairs of observations in which predicted ranking of survival probabilities corresponds to actual ranking. Calibration is a measure of the accuracy of predicted probabilities that compares expected to actual outcomes. The combined registry population consisted of 62,955 patients and 5.1% of patients underwent revision surgery during a mean follow-up time of 7.6 ± 4.5 years. The best performing models had concordance in the moderate range (0.67, 95% CI 0.64–0.70) and were well calibrated at all follow-up times. Multiply imputed data did not show notable differences from the complete case analysis. Machine learning analysis of the combined registers enabled the prediction of subsequent revision surgery risk after primary ACLR with moderate accuracy. However, the most important finding of this study was that this analysis of nearly 63,000 patients yielded similar prediction accuracy as a previous study of approximately 25,000 patients. This suggests a so-called ceiling effect of the registers has been reached and that simply adding more patients to the database is unlikely to appreciably improve prediction accuracy. For an improvement in the ability to predict outcome based on knee ligament register data, an evolution in the variables collected is required. This represents a significant challenge as the balance between optimal variable collection and surgeon compliance is a delicate one; data collection must be streamlined to avoid survey fatigue and the addition of variables to the registry must be carefully considered, weighing the added value against the additional burden on the surgeons which may compromise compliance. Less onerous methods of improving data collection such as natural language processing may hold the key to improving outcome prediction using the knee ligament registers.