Adapting to Marginal Distribution Shifts in Supervised Learning: A Double-Weighting Approach

Supervised classification traditionally assumes that training and testing samples are independently and identically distributed (i.i.d.) from the same underlying distribution. However, practical scenarios are often affected by distribution shifts, such as covariate shift and label shift. In covariate shift, the marginal distribution over the instances (covariates) differs at training and testing while the label conditional distribution remains the same. In label shift, the marginal distribution over the labels differs at training and testing while the instance conditional distribution remains the same. Additionally, in multi-source scenarios, the training data is obtained from multiple sources, each of which has different probability distributions. In scenarios affected by distribution shift, conventional supervised classification methods, like empirical risk minimization, can perform poorly because the empirical risk approximates the training expected risk, rather than the testing expected risk. Most existing techniques for correcting distribution shifts are based on a reweighted approach that weights training samples, assigning lower relevance to the samples that are unlikely at testing. However, these methods may achieve poor performance under support mismatch or when the weights obtained take large values at certain training samples. In addition, in multi-source cases, existing methods inherit the problems of single-source reweighted methods and do not exploit complementary information among sources, equally combining sources for all instances. In this dissertation, we establish learning methodologies for supervised learning under marginal distribution shifts. The methodology proposed is based on minimax risk classification and avoids the limitations of existing methods by weighting both training and testing samples. For the multi-source case, the presented methods assign source-dependent weights for training and testing samples, where weights are obtained jointly using information from all sources. In addition, we develop effective techniques that obtain the sets of training and testing weights, generalizing the techniques based on the conventional kernel mean matching. We also present generalization bounds for the proposed methods that show a significant increase in the effective sample size. Empirically, the proposed methods achieve enhanced classification performance in both synthetic and empirical experiments using multiple real datasets. This dissertation makes theoretical contributions leading to efficient algorithms for multiple supervised learning scenarios under distribution shifts with classification rules that provide confidence in the predictions and enhanced performance in comparison with state-of-the-art techniques.