Comparison of parametric versus machine-learning multiple imputation in clinical trials with missing continuous outcomes

Abstract Background Flexible machine-learning (ML) models to generate imputations within the Multiple Imputation (MI) framework has recently gained traction, particularly in non-randomised observational settings. For randomised controlled trials (RCTs), it is unclear whether ML approaches to MI, in combination with Rubin’s Rules, provide valid inference in terms bias, confidence interval coverage, mean squared error (MSE), Type I error and power of the treatment effect estimator. Methods We conducted two simulation studies in RCT settings that have an incomplete continuous outcome but fully observed covariates and treatment assignment. We compared Complete Cases, standard MI (MI-norm), MI with predictive mean matching (MI-PMM) and ML-based approaches to MI, including classification and regression trees (MI-CART), Random Forests (MI-RF) and SuperLearner when outcomes are missing completely at random or missing at random conditional on treatment/covariate. The first simulation explored a cross-sectional outcome with non-linear covariate-outcome relationships in the presence/absence of covariate-treatment interactions. The second simulation explored skewed repeated measures, motivated by a trial with digital outcomes. Results For the cross-sectional simulation without interaction, we found that Complete Cases yielded valid inference; MI-norm performed similarly, except when there is a non-linear covariate-outcome relationship and missingness depends on the covariate. ML approaches led to smaller MSE in specific non-linear settings, but provided unreliable inference for others. MI-PMM, which is the default setting in the package , led to unreliable inference under some settings. In the presence of complex treatment-covariate interactions, performing MI separately by arm, either with MI-norm, MI-RF or MI-CART, provided inference that had comparable or better properties compared to Complete Cases when the analysis model omits the interaction term. In the repeated measures setting, ML approaches to MI and MI-PMM led to bias and under- or over-coverage, particularly when missingness depended on treatment. Conclusion Based on the simulation findings, Complete Cases and MI-norm are more appropriate than ML approaches to MI for making inference, especially for late phase RCTs where Type I error control is crucial. While ML approaches may provide gains in MSE for complex covariate-outcome relationships, results should be interpreted with the caveat that Rubin’s Rules are not guaranteed to be valid when used with ML imputation methods, and can lead to bias in the estimated effect and/or its standard error.