Qualified prediction system for allograft failure in real world settings: extended validation study

Objective To perform comprehensive validations of the integrative Box (iBox) system, a prediction model for long term risk of kidney allograft failure, for extension of its context of use in clinical trials as well as for its wider implementation in clinical practice. Design Extended validation study. Setting Paris Transplant Group database (comprising kidney recipients with transplantations between 1 January 2005 and 1 January 2014) and European, North American, and South American hospitals (comprising recipients of kidneys transplanted beween 1 January 2000 and 1 January 2022). Patients were followed until 1 November 2024. Participants 12 683 kidney tranplant recipients from 21 academic centres in Europe, North America, and South America; 4000 patients in the derivation cohort and 8683 in the validation cohorts. Main outcome measures Performance of the iBox, including flexible iBox versions in specific clinical contexts (race-free estimated glomerular filtration rate (eGFR) equations (ie, without including race as a factor in the calculation), in specific clinical contexts (initial nephropathy recurrence, BK virus associated nephropathy, and different immunosuppressive strategies), and over-extended follow-up periods. Predictive performance was assessed by discrimination, calibration, overall fit, and clinical utility. Results 12 683 kidney transplant recipients were included in the study (n=4000 in the derivation cohort and n=8683 in the validation cohorts). Median follow-up time after risk evaluation was 5.78 years (interquartile range (IQR) 3.51-7.00) in the derivation cohort and 4.68 years (2.48-7.00) in the validation cohorts. 549 (13.7%) and 991 (11.4%) patients had graft loss in the derivation and validation cohorts, respectively. All versions of the iBox algorithm maintained good discrimination and overall fit performance in the derivation and validation cohorts (C index range 0.79-0.87, Brier scores 0.08-0.11). Calibration was adequate in some but not all external validation cohorts, with trends toward overestimation or underestimation of predicted risks. Decision curve analysis showed positive and comparable net benefit for all iBox algorithms across decision thresholds up to 40% in the derivation cohort (net benefit 0.07-0.08 at 20% threshold) and validation cohorts (net benefit 0.03-0.11 at 20% threshold). Accounting for the competing risk of death with a functioning graft resulted in similar performance, except for calibration which varied across cohorts, without any model consistently outperforming any other model. The model performed well with different race-free eGFR equations (C index 0.81), in various clinical scenarios, including disease recurrence and BK virus nephropathy, with different immunosuppressive strategies, such as calcineurin inhibitors and mTOR (mechanistic target of rapamycin) inhibitors (C index range 0.74-0.87), and when extending the prediction period to 10 years after risk evaluation (C index 0.79). The iBox predictive performance was not modified when various histological indices were used. The iBox was also superior to eGFR slope (C index 0.81 v 0.62) and circulating anti-HLA donor specific antibodies (C index 0.81 v 0.57) in its predictive ability. Conclusions In this study, the robust predictive performance of the iBox system across diverse real world settings and clinical scenarios was shown. These results highlight the versatility and reliability of the iBox system, and support its use for risk stratification in routine clinical practice and as a surrogate endpoint for clinical trials.