Optimizing Neural Network Loss Function with Surrogate-assisted Differential Evolution

The cross-entropy loss is widely used in neural networks for classification problems, however it may not be the best choice. In this study the problem of designing new loss function is considered, in particular the Taylor polynomial is applied. To tune the parameters of the polynomial, differential evolution is used. As the fitness evaluation requires training a neural network, it is a computationally expensive process. To overcome this problem, an approach based on surrogate models is used, in particular the Kriging method applied. Experiments on the CIFAR-10 test set have shown that the surrogate approach accelerates the search for the optimal solution, and also allows achieving better accuracy in comparison with the cross-entropy or the coefficient optimization method without using the surrogate approach. The found loss function curve has significantly different shape compared to the cross-entropy, and has two minima at probability of correct prediction equal to 0 and 1.