A COMPARATIVE STUDY OF ACTIVATION FUNCTIONS IN DEEP LEARNING MODELS

Activation functions play a vital role in the training dynamics and generalization performance of deep learning models. This study presents a comparative analysis of ten widely used activation functions—ReLU, Sigmoid, Tanh, ELU, SELU, Softplus, Softsign, Swish, GELU, and a custom spline-based function—within a unified convolutional neural network (CNN) architecture. All models were trained and evaluated on the CIFAR-10 dataset under identical experimental settings, including fixed learning rate, batch size, number of epochs, and architecture configuration. The results show that the proposed spline activation function achieved the highest test accuracy of 71.48%, outperforming popular functions such as ReLU (67.87%) and Swish (68.33%). In contrast, traditional functions like Sigmoid exhibited significantly lower accuracy (10.00%), reaffirming known limitations in deep network contexts. These findings demonstrate the potential of adaptive, piecewise-defined activation functions to enhance model performance while maintaining competitive training efficiency. The study provides practical insights into activation function selection for image classification tasks.