An Algorithm for the Construction of Improved Training Sets for Neural Networks

In this paper we represent a novel approach for improving the training process of an arbitrary backprop-based (e.g. convolution) neural networks, in terms of accuracy and speed. The gain in the performance is due to the alternative method of selection of the samples, to be included in the training set. In the presented Entropy Maximal (EntMax) approach, the samples having higher relative entropy are to be selected to comprise the training set. The images of handwritten characters are considered as discrete random independent and identically distributed (i.i.d.) variables. By applying this approach one takes advantage not only by improving the prediction ability of the neural network, over the validation set, but also by decreasing the time necessary for training, due to the lower number of samples, necessary to provide satisfactory classification ability and the lower number of epochs, needed to assure satisfactory level of the error rate.