Finding the Sweet Spot: Speed vs. Fidelity in Encrypted LLMs via Proxy Simulation

Fully homomorphic encryption (FHE) enables arithmetic operations to be performed directly on ciphertext, but this encrypted arithmetic is orders of magnitude slower than plaintext. When securing a large language model (LLM) using the CKKS encryption scheme, the slowdown is compounded by the need to replace each non-linear layer with a polynomial, which also injects approximation noise that can erode accuracy. Higher-degree polynomials reduce this error yet increase multiplicative depth and latency. Therefore, designers need to pinpoint where additional depth no longer pays off. To this end, we introduce a lightweight noise-injection proxy that measures CKKS residuals once, stores them, and replays the noise inside the plaintext model while sweeping polynomial degrees. Our proxy produces an accuracy-versus-latency curve and reveals the "sweet spot": The polynomial degree beyond which extra depth yields only marginal accuracy gains for a steep time cost. This curve guides developers in choosing an appropriate polynomial degree, eliminating the need for time-consuming encryption benchmarks.