Homomorphic approximation of non-linear activation functions using the CKKS cryptosystem of the OpenFHE library

Abstract Homomorphic computing can be viewed as the holy grail of encryption, as it allows evaluating computations on encrypted data. This enables perfect privacy for cloud applications, as a cloud provider handling user data never gets to see the data it is performing calculations on. Unfortunately, homomorphic cryptosystems can only evaluate a limited number of mathematical functions. One of the most promising cryptosystems today is the CKKS cryptosystem, which allows the evaluation of encrypted multiplications and additions. An essential part of neural networks and machine learning, however, are activation functions that are purposely built to introduce non-linearity into the neural network and, therefore, can not be evaluated easily using only multiplications and additions. Solving this problem is currently done by evaluating approximation polynomials homomorphically. However, this method is imprecise and often requires modifications to the training process and extensive manual optimization of the network to be feasible. In this article, we propose a novel way of implementing the eight commonly used activation functions Binary-Step, ReLU, parametric ReLU, Sigmoid, Tanh, GELU, ELU and Swish, by building them with the help of the homomorphic square-root, inverse, and exponential evaluation methods proposed by Prantl et al. We conducted an extensive parameter study, helping developers to choose parameters suiting their requirements. Our activation functions mostly outperformed the classical approach of polynomial approximation on smaller evaluation intervals and, for some functions, also for larger intervals in both accuracy and computational efficiency. We found that both the popular ReLU and more recent GELU activation functions were among the most well-suited activation functions for homomorphic evaluation.