Comparison of Predictive Probability Calculation Methods for Bayesian Interim Monitoring with Delayed Binary Responses: A Simulation Study of the Handling of Pipeline Subjects

Abstract Background In Bayesian interim monitoring with delayed responses, the handling of pipeline subjects whose outcomes remain pending at the time of interim analysis may affect predictive probability (PP) estimates and interim decision-making. To investigate this, we considered a single-arm Bayesian phase II trial with delayed binary responses and compared five approaches for handling pipeline subjects. Methods Available-case (AC), all-nonresponse (ANR), all-response (AR), fixed-rate plug-in (FRP), and observed-rate plug-in (ORP) methods were compared. Maximum sample size was 40, with 20 subjects whose outcomes had been ascertained at the recruitment decision point, at which a PP-based interim early-efficacy decision was made. We considered 36 response-rate scenarios, resulting in 72 analysis conditions with five or 10 pipeline subjects. Binary outcomes for subjects with ascertained outcomes and pipeline subjects were generated from Bernoulli distributions with a common true-response probability. Reference PP was used as a retrospective oracle benchmark. Performance was evaluated using the Pearson correlation coefficient between the PP obtained using each method and the reference PP, bias, mean absolute error (MAE), root mean square error (RMSE), and agreement with the reference PP–based interim decision. Results AC and ORP performed favorably across several metrics. Median coefficients for five pipeline subjects were 0.815 for AC and 0.807 for ORP; for 10, they were 0.647 and 0.661. Across the 72 analysis conditions, mean bias was − 0.018 for AC and − 0.008 for ORP. MAEs were 0.059 and 0.055; RMSEs were 0.098 and 0.103. Mean decision agreement for five pipeline subjects was 0.914 for AC and 0.938 for ORP; for 10, they were 0.872 and 0.919. For all methods, increasing pipeline subjects from five to 10 resulted in lower median Pearson correlation coefficients and lower mean decision agreement. Conclusions ORP had the mean bias closest to zero, lowest mean MAE, and highest mean decision agreement; AC had the lowest mean RMSE. AC and ORP showed relatively high Pearson correlation coefficients with the reference PP. These findings indicate that the performance of approaches to handling pipeline subjects varies and should be assessed using multiple measures, including association with the reference PP, estimation error, and interim decision-making agreement.