A Data Pruning Method with Feature Distillation for Improved Computational Efficiency

As the capabilities of Neural Network models grow, so does the cost of power consumption and time, which consequently makes training state of the art models on resource constraint devices more challenging. In this paper, we present a novel method to reduce training costs by performing a version of data pruning, which prunes batches of data based on how far their feature extractions are from a set of feature distilled vectors. We call this new method Data pruning via Feature distillation. We employ a neural network layer, referred to as a feature distilled layer, which maintains a set of vectors which aim to approximate the distribution of extracted features within a neural network. This set of feature distilled vectors dynamically adjusts throughout the training process to accomplish this approximation while also adjusting to changes in the network throughout training. We demonstrate that this method significantly reduces overall training time while maintaining or even improving model performance across various datasets and architectures. Experiments conducted on MNIST, CIFAR10, CIFAR100, and CalTech-256 using ResNet models show performance increases of up to 47.41 % speed up and a 40.11 % decrease in GPU power consumption, while also maintaining accuracy within 1 % of the baseline model. Our work presents as a flexible addition to a neural network which can be added in after feature extraction layers to be used for accelerating training through data pruning.