ReLU-like non-monotonic smooth activation functions based on regularized heaviside functions and extensions

ReLU is a widely adopted activation function in neural networks of deep learning for different application tasks. In order to overcome its limitations of non-smoothness and dying neuron problems, various improved versions with non-monotonic, smooth properties have been proposed via different innovative methods. In this work, we present a systematic method to construct this kind of activation functions via regularized S-shaped Heaviside functions based on exponential function, logarithmic function, trigonometric function, and Gaussian error function, etc.. The derived ReLU-like activation functions have the common properties of nonlinearity, infinite differentiability, non-monotonicity, zero-centering. Based the requirements of S-shaped regularized Heaviside functions, we modify some existing classical Logish, Smish, and CaLU to ReLU-like activation functions to approach ReLU at positive or negative regions to inherit the merits of ReLU. Additionally, we extend the proposed method to activation functions composed of two and three piecewise linear functions, and present some examples, such as Leaky ReLU, V-shaped ReLU, projection formula, soft and hard thresholding formulas. This method can be utilized to transform the activation functions with multiple piecewise linear functions into regularized ones naturally.