Agreement between heuristic shrinkage factor and optimal shrinkage factors in logistic regression for risk prediction: a simulation study across different sample sizes and settings

Introduction: The heuristic shrinkage factor of Van Houwelingen and Le Cessie (SVH) is a commonly used closed-form solution to adjust for overfitting in unpenalised logistic regression models for risk prediction. It is also the basis of widely-adopted minimum sample size criteria for developing clinical prediction models. However, current evidence is lacking regarding the bias of Sv compared to the optimal shrinkage factor (Sopt). Here, we examine this issue and also assess the bias of an alternative bootstrap-derived shrinkage factor (Sboot).<br/><br/>Methods: We undertook two simulation studies. The first examined the bias of Sun and Sboot as estimators of Sopt across a range of different scenarios defined by Cop, the C-statistic of the model developed in a population sized dataset. The second examined the bias of Sopt when using development sample sizes targeting a shrinkage of 0.9, based on a sample size calculation defined by Svu itself (Noriginal) or by an adapted simulation-based approach (Nsim).<br/><br/>Results: For high C-statistics, SVH overestimated Sopt, whereas for low C-statistics SVH underestimates Sopt. For example, across scenarios when 0.8 ≤ Cpop &lt; 0.85, the 95-percentile range in the bias was (0.005, 0.387), compared to (-0.580, -0.003) across scenarios when 0.6 ≤ Cop &lt; 0.65. The magnitude of bias increased as Cpop tended to either 0.5 or 1. As sample size increased and Sopt → 1, the magnitude of the bias in either direction reduced. Sboot was less biased than SVN, with a median bias across all scenarios of 0.007 compared to 0.032 for SVH. Developing models on datasets of size Nsim gave mean (Sopt) closer to 0.9 (mean magnitude of difference across all scenarios 0.004) than Noriginal (mean magnitude of difference 0.038).<br/><br/>Conclusions: SVH is often a poor estimator of the optimal global shrinkage factor. If global shrinkage is needed, we rather recommend using the bootstrap shrinkage estimate. The bootstrap estimate shows minimal bias in most scenarios, though in small samples the variability is large so provides no guarantees to address overfitting in a single dataset. A sample size calculation based on simulation is often preferable over SVH, but convergence issues can arise with c-statistics &lt; 0.7.