Statistical inference in classification of high-dimensional Gaussian mixture

Abstract We consider the classification problem of a high-dimensional mixture of two Gaussians with general covariance matrices. Using the replica method from statistical physics, we investigate the asymptotic behavior of a broad class of regularized convex classifiers in the limit where both the sample size n and the dimension p approach infinity while their ratio <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>α</mml:mi> <mml:mo>=</mml:mo> <mml:mi>n</mml:mi> <mml:mrow> <mml:mo>/</mml:mo> </mml:mrow> <mml:mi>p</mml:mi> </mml:mrow> </mml:math> remains fixed. This approach contrasts with traditional large-sample theory in statistics, which examines asymptotic behavior as <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" overflow="scroll"> <mml:mrow> <mml:mi>n</mml:mi> <mml:mo stretchy="false">→</mml:mo> <mml:mi mathvariant="normal">∞</mml:mi> </mml:mrow> </mml:math> with p fixed. A key advantage of this asymptotic regime is that it provides precise quantitative guidelines for designing machine learning systems when both p and n are large but finite. Our focus is on the generalization error and variable selection properties of the estimators. Specifically, based on the distributional limit of the classifier, we construct a de-biased estimator to perform variable selection through an appropriate hypothesis testing procedure. Using L 1 -regularized logistic regression as an example, we conduct extensive computational experiments to verify that our analytical findings align with numerical simulations in finite-sized systems. Additionally, we explore the influence of the covariance structure on the performance of the de-biased estimator.