Arctangent-based nonlinear activation unit - ATLU

In the current most popular field of deep learning, activation function refers to a mathematical function applied to perform non-linear transformation on the input. At present, it is commonly used in the hidden and output layers of feature extraction networks. When activation functions are introduced, the non-linear capabilities of neural networks are enhanced, Meanwhile, the model performs better in terms of expressive power and fitting ability. Inspired by trigonometric functions, an activation function is proposed in this study. Based on the inverse tangent trigonometric function -, namely ATLU, which is the arctangent-based nonlinear activation unit, it is expressed as <i>f(x)</i> = <i>x(arctan(1.09x) + <i>&pi;</i>/2)</i>. This activation function can be applied to address the gradient vanishing and exploding in deep neural networks. It is validated on the cifar100 dataset with the VGG11 network as the benchmark.