Multivariate activation functions

Activation functions are critical components in neural network architectures, significantly influencing model performance and learning efficiency. While traditional univariate activation functions have been extensively studied and optimized, the exploration of multidimensional (multivariate) activation functions remains relatively nascent. In this paper, a comprehensive study of various multidimensional activation functions is presented, focusing on both Gaussian Mixture Models (GMM) and interpolation-based methods with learnable parameters. Through extensive experiments, the performance of these novel activation functions is evaluated in diverse neural network architectures and tasks. Results demonstrate that while the proposed multidimensional activation functions achieve performance on par with known activation functions like Swiglu, they do not consistently outperform it. This investigation provides valuable insights into the potential and limitations of multidimensional activation functions, paving the way for future research in this promising area of deep learning.