SRS235 - Assessment and mitigation of subgroup underperformance in machine learning models for kidney transplant outcome prediction

Abstract Introduction Machine learning (ML) models are increasingly being developed to predict outcomes in patients undergoing kidney transplantation. Little attention has been paid to assessing their performance across diverse patient subgroups, with risk of bias in predictive performance in subgroups of patients with protected characteristics. This study aimed to evaluate and address potential underperformance of a graft and patient survival prediction model in clinically relevant patient subgroups. Methods We assessed the predictive accuracy of a DeepHit survival analysis model, trained on over 25 years of transplant outcome data. The model's performance was evaluated across subgroups defined by patient ethnicity, sex, age, and pre-transplant sensitization status. To mitigate identified performance disparities, we applied several strategies to re-balance the model's training process to ensure all patient populations were adequately considered. Results Analysis revealed that the model's performance for graft survival was consistent across all investigated subgroups. However, a significant disparity was observed in patient survival predictions. The model demonstrated notably worse predictive accuracy for Black patients and patients over 70 years old, indicating a clear underperformance in these specific populations. Using bias mitigation strategies, we were able to partially close this performance gap. Conclusions This study highlights the critical importance of evaluating ML models for subgroup-specific underperformance in a clinical context. Our findings underscore that while overall model performance may appear strong, it can mask significant disparities. The application of mitigation strategies is essential for developing equitable and reliable decision support tools to ensure fair outcomes for all patients.