Neural network optimization via inactive neuron removal and neuron merging in single input single output architectures

In this paper, a novel optimization algorithm, namely the JeevanRuth neural network optimization algorithm (JRNNOA), is presented for feedforward neural networks (FNNs) with a single hidden layer (HL), single input and single output, explicitly focusing on architectures utilizing the rectified linear unit (ReLU) activation function. The proposed methodology systematically eliminates inactive neurons and integrates neurons with similar behaviours, effectually mitigating the size of the network while preserving its functional integrity. A comprehensive algorithm analysis is provided, with theoretical justification and discussion in the context of the Universal Approximation Theorem. Extensive investigational results demonstrate a significant reduction in neurons without compromising performance, suggesting potential applications in resource-constrained environments.