A learnable activation function based on orthogonal Legendre wavelets

Activation functions play a crucial role in introducing nonlinearity into neural networks. Previous approaches, such as rectifying neurons and logistic sigmoid neurons, primarily focused on constructing activation functions based on biological models. These methods have achieved considerable success in various machine learning tasks. However, they are inadequate in addressing two critical issues: Gradient Vanishing and Dead ReLU. This research demonstrates that employing Legendre orthogonal wavelets can effectively alleviate these issues and yield comparable or superior performance compared to rectifying neurons. In this study, the input features prior to the activation layer are normalized to the range [Formula: see text]. Each mini-interval within this range is then mapped using a set of orthogonal Legendre wavelets. Subsequently, the activation function produces a weighted sum of all the mini-intervals. By modifying typical models used in image classification, this paper found that incorporating Legendre wavelets resulted in significant improvements in network accuracy. With minor adjustments in the hidden layers, ResNet50 achieved an increase of +5.17% in accuracy for CIFAR10 classification. Similarly, in CIFAR100 classification, the use of Legendre wavelets in Swin Transformers led to a +3.51% boost in accuracy.