Learning Kolmogorov-Arnold Neural Activation Functions by Infinite-Dimensional Optimization

Inspired by the Kolmogorov-Arnold Representation Theorem, Kolmogorov-Arnold Networks (KAN) have recently reshaped the landscape of functional representation in the context of training univariate activation functions, thus achieving remarkable performance gains over traditional Multi-Layer Perceptron (MLPs). However, the parametrization of the activation functions in KANs hinders their scalability and usage in machine learning applications. In this article, we propose a novel infinite-dimensional optimization framework to learn the activation functions, enabling the achievement of boundedness, interpretability, and, seamless compatibility with training by backpropagation are all preserved for the network, therefore, reducing the number of required trainable parameters. Through functional representation examples, our results reveal superior accuracy in tasks such as functional representation.