Pre-train. Mixup and Fine-tune: A Simple Strategy to Handle Domain Shift

Transfer learning leverages models trained on large source datasets to target domains with limited datasets by fine- tuning pre-trained models. These approaches work well with minimal distribution shifts. However, the nature of distribution shift is unknown in real-world applications which may lead to worse performance of models in the field. Domain adaptation approaches handles this issue explicitly by adapting source trained models using discrepancy, adversarial or reconstruction based approaches, but doesn't tackle the limited target data issue. A data augmentation approach such as mixup helps train resilient models with limited datasets, and is recently being considered for its ability to handle distribution shifts. In this work, we investigate how mixup can be used along with transfer learning to improve model performance on target domains with distribution shifts. Our experimental results shows mixup is complementary to transfer learning which we demonstrate by varying the percentage of available target training data with publicly available source and target datasets. Our proposed approach of using mixup along with fine tuning shows improved performance than just fine tuning or mixup across varying percentage of target dataset sizes.