Adaptive L1 Regularization for Neural Network-Based Symbolic Regression

Symbolic regression provides an analytical method to derive explicit mathematical relationships from empirical data that elucidate underlying processes. The model produced aims to be interpretable and as reliable as a physical law. In this context, a neural network architecture named Equation Learner (EQL) has been crafted to formulate equations via a fully differentiable system. Traditionally, the clarity and precision of EQL predictions are maintained through a hybrid strategy that initially sets to 0 the model parameters' L1 regularization to maximize accuracy, then activates the L1 regularization later in training to drive the model towards structural simplicity. This paper identifies weaknesses in this regularization approach and introduces a continuous adaptive L1 regularization technique, we have named L1adapt. This new approach progressively adjusts the level of constraints on the model weights, finely tuning the penalty intensity in relation to the model’s accuracy as training advances.