The Authors Respond

To the Editor: We appreciate Williams et al.’s recommendation of increasing cross-fitting folds to decrease variability due to random seed for cross-fit estimators.1 The authors clearly demonstrate the efficacy of this solution for the cross-fit estimator of the average treatment effect in our simulation scenario. Despite these findings, we still encourage caution and additional research in this area. We re-ran the code provided by Williams et al. with different numbers of cross-fitting folds and calculated relevant performance measures for each estimator (Table). We observe a clear trend of increasing variability with the number of cross-fitting folds, with 2.4-fold higher variability for the 40-fold cross-fit estimator compared with the two-fold cross-fit estimator. In this particular simulation, the variance of all estimators is small relative to the bias, and thus the mean squared error of the two-fold versus 40-fold estimators are roughly the same. Nevertheless, these results suggest that there is a bias-variance tradeoff involved when selecting the number of folds, which is consistent with our observations in previous work.2 Thus, increasing the number of cross-fitting folds specifically for the purpose of reducing random seed variability may not be the optimal approach in practice. Further research may be needed to identify a set of concrete guidelines that best address both bias-variance tradeoffs and random seed variability. TABLE. - Operating Characteristics of Cross-fit Augmented Inverse Probability of Treatment Weights Estimators of the Average Treatment Effect With Different Numbers of Cross-fitting Folds (CV-Folds), Based on One Seed and Averaging Over 100 Seeds Metric CV-Folds One Seeda Average Over 100 Seeds Bias 2 −0.08328 −0.08336 10 −0.07877 −0.07876 20 −0.07826 −0.07827 40 −0.07813 −0.07798 Variance 2 0.00049 0.00042 10 0.00099 0.00095 20 0.00108 0.00108 40 0.00117 0.00118 Mean squared error 2 0.00742 0.00737 10 0.00719 0.00715 20 0.00720 0.00720 40 0.00727 0.00725 aOperating characteristics based on the first analysis result for each of the 200 datasets analyzed. For now, we maintain our recommendation that analysts should be aware of and routinely explore sensitivity of results to random seed, particularly when analyzing small data sets. Random seed dependence will vary by analysis and analytic method, and what is considered an “important” difference between results across seeds will depend on the application area. If substantial variability is found through sensitivity analyses, averaging may still provide a quick solution that is agnostic to the original analysis strategy.