OLA: An FPGA-based <u>O</u>ver<u>l</u>ay <u>A</u>ccelerator for Privacy Preserving Machine Learning with Homomorphic Encryption

Homomorphic Encryption (HE) is a promising technique for protecting user privacy in cloud-based Machine Learning (ML) inference. However, homomorphically encrypted operations are orders of magnitude slower than the corresponding unencrypted operations due to high computation and memory bandwidth requirements. FPGAs are attractive platforms for designing domain-specific accelerators. Manually programming FPGAs for HE ML inference is challenging, as different HE parameters and operations require different mapping strategies. We propose OLA, the first FPGA-based overlay accelerator for low latency computation on homomorphically encrypted data. OLA eliminates the need for manual and time consuming FPGA programming by providing a Python-based interface and a compiler that efficiently maps OLA programs to hardware instructions for execution. The hardware architecture and instruction set are co-designed to accelerate common HE primitives, while the compiler maps all the required HE operations to these primitives for efficient execution. OLA features two optimizations to address memory bandwidth challenges in HE computation. First, we propose an asynchronous dataflow execution model where the compiler manages data processing order and the architecture enforces it at run time. This approach enables guaranteed data reuse via on-chip SRAMs. Second, we design latency-aware instruction scheduling in the compiler to reduce data reuse distance. We implement the overlay accelerator on the AMD Alveo U280 FPGA. We evaluate the effectiveness of the proposed overlay accelerator by executing various HE operations, HE linear algebra benchmarks, and end-to-end ML inference. Experimental results show that OLA reduces the latency of HE operations by up to 683× (4.4×) compared with State-Of-The-Art (SOTA) CPU (GPU) implementations. It achieves up to 3212× speedup in latency compared with the SOTA CPU implementations for HE ML inference.