Minimax Optimal Estimation of Stability Under Distribution Shift

How to Efficiently Estimate Stability Measures: A Minimax Optimal Approach In “Minimax Optimal Estimation of Stability Under Distribution Shift,” Hongseok Namkoong, Yuanzhe Ma, and Peter W. Glynn address the challenge of benchmarking the performance of decision policies and prediction models under distribution shift. Conventional risk measures and distributionally robust losses typically require specifying the magnitude of possible distribution shift—a quantity that is difficult to determine in practice. Instead, the authors consider a stability measure defined in terms of the acceptable level of performance degradation, which is more intuitive. To efficiently estimate this measure, they consider an estimator based on the dual formulation of the stability measure and show that this estimator is minimax optimal. Their results quantify the convergence rate of the estimator, which exhibits a fundamental phase shift behavior. In addition, they empirically observe that the stability measure reliably captures system performance under distribution shift in applications including queueing systems and healthcare prediction tasks.