Finding an Optimal Small Sample of Training Dataset for Computer Vision Deep-Learning Models

While the typical approach in deep learning is “more data, better model,” in this work, we would like to explore an alternative setting where the training phase has limited resources in terms of time, memory, or computing, and we cannot use a large training dataset but only a radically small sample of it. For such a setting, we present a novel method to select a subset of the training instances that outperform naive random sampling and produce a higher model accuracy. We analyze the proposed method's running time to show its efficiency, and provide a simulation study over three benchmark datasets (MNIST, CIFAR-10, and SVHN) to demonstrate its advantage compared to random sampling. The proposed method derives significantly better models, especially for radically small training sizes, reaching a peak of hundreds of percentage improvement for the MNIST dataset, 31% for CIFAR-10, and 11% for SVHN.