Lightening Encrypted Convolutions: A GPU-Optimized Approach to Private Inference

Deep learning models, particularly those used for im-age classification, have become pervasive in today's technological landscape. However, the increasing demand for privacy, espe-cially when sharing data with external parties, poses significant challenges. Homomorphic encryption (HE) offers a compelling solution by enabling computations on encrypted data without compromising privacy. Despite its potential, the widespread adoption of HE remains limited due to its computational complexity. In this work, we propose an optimized approach to improve the efficiency of a fundamental building block in image classification models: the convolution operation. Our approach is based on the observation that multiple kernel evaluations in a convolution layer can be performed in parallel, provided that their receptive fields do not overlap. Using this property, we reorganize convolution computations into independent groups, significantly reducing redundant operations. This grouping reduces the number of multiplications by up to 75% compared to standard encrypted convolution techniques. Additionally, the approach eliminates the need for costly preprocessing, enabling direct processing of encrypted inputs. To further accelerate homomorphic operations, we leverage a GPU-accelerated HE library, enabling faster op-erations. These contributions make encrypted convolution signifi-cantly more practical for real-world deep learning applications.