Refining Neural Network with Symbolic Regression

Classical neural network (NN) compression methods, such as NN pruning and distillation, focus on reducing the NN size. However, these methods often sacrifice accuracy in the process. To address this issue, we propose a novel NN refinement method based on symbolic regression called SR-R. At its core, SR-R employs a symbolic regression method based on Cartesian genetic programming to find a compact mathematical expression that accurately approximates the input-output relationship of a selected module within the NN. SR-R then replaces the targeted module with the discovered mathematical expression. Finally, SR-R fine-tunes the parameters of the remaining modules in the NN. Experiments are conducted on two types of NN benchmarks: multilayer perceptron NNs (MLP) and convolutional NNs (CNN). Experimental results show that, compared with NN pruning and NN distillation, SR-R can effectively reduce the number of NN parameters while maintaining or even enhancing inference accuracy.