Cox Reduction and Confidence Sets of Models: A Theoretical Elucidation

For sparse high-dimensional regression problems, Battey and Cox (Proc. R. Soc. Lond. A 474 (2018) 20170631) and Cox and Battey (Proc. Natl. Acad. Sci. USA 114 (2017) 8592–8595) emphasised the need for confidence sets of models: an enumeration of those small sets of variables that fit the data equivalently well in a suitable statistical sense. This is to be contrasted with the single model returned by penalised regression procedures, effective for prediction but potentially misleading for subject-matter understanding. The proposed construction of such sets relied on preliminary reduction of the full set of variables. While various possibilities could be considered for this, Cox and Battey (Proc. Natl. Acad. Sci. USA 114 (2017) 8592–8595) proposed a succession of regression fits based on incomplete block designs. The purpose of the present paper is to provide some new insight into that work. For an unspecified reduction strategy, we begin by characterising models that are likely to be retained in the model confidence set, emphasising geometric aspects. We then evaluate possible reduction schemes based on penalised regression or marginal screening, before theoretically elucidating the reduction of Cox and Battey (Proc. Natl. Acad. Sci. USA 114 (2017) 8592–8595). We identify features of the covariate matrix that may reduce its efficacy, and indicate improvements to the original proposal. An advantage of the approach is its ability to reveal its own stability or fragility for the data at hand.