Smoothed Multithreshold Activation Functions in the Learning of Neural Networks

The choice of the activation function significantly affects the convergence rate and stability of the learning of neural networks, as well as their ability to approximate complicated functions. We consider a model of smoothed multithreshold function (SMTF), which allows more accurate modelling of nonlinear dependencies and improves the efficiency of the network training compared to usual sigmoid or rectifier-like activations. The paper concerns theoretical analysis of properties of two kinds of SMTF and their comparison with classical and modern activation functions such as logistic sigmoid, tanh, ReLU, Swish, etc. The effectiveness of SMTFs as a neuron activation was empirically estimated through series of experiments consisting of the approximation of a complex nonlinear mapping. A comparative analysis of obtained results demonstrates that SMTF-based activations provide better accuracy and are able to catch dynamic data changes that is important for problems where traditional activation functions demonstrate limited efficiency. Furthermore, the use of SMTFs can reduce the number of epochs required to train neural models properly.