Training by Pairing Correlated Samples Improves Deep Network Generalization

Deep neural networks (DNNs) have been widely applied in different application domains. The DNN was first studied intensively in vision applications before adapting it to other fields. To migrate DNN solutions from the vision domain to another application domain, a neural network solution may be influenced by new structures or improved training processes. This article focuses on training process improvements. We propose a pairing technique that simultaneously optimizes the performance of the models on pairs of training samples during the training process. A pair includes an original training example and a corresponding modified version. Our pairing techniques adapt the similarity part of the contrastive loss as an additional regularization term for the loss function of a given machine learning task. The pairing techniques show at least 1% improvement in network accuracy on top of mix-up augmentation for the CIFAR10 dataset and 2% increase in accuracy for DCASE 2020 Task 1A data. We show that the proposed training-by-pairs provides parameter regularization for ReLU deep networks. As a result, the technique can potentially apply to many other machine learning applications.