BlindTuner: On Enhancement of Privacy-Preserving Fine-Tuning of Transformers Based on Homomorphic Encryption

Fine-tuning pretrained models has emerged as a pivotal technique in machine learning, especially in situations with limited, task-aligned training data. However, challenges surface when data sharing encounters obstacles due to stringent privacy regulations or user apprehension regarding personal data disclosure. Earlier works based on secure multiparty computation (SMC) and fully homomorphic encryption (FHE) for privacy-preserving machine learning (PPML) focused more on privacy-preserving inference than privacy-preserving training. While privacy-preserving transfer learning has made progress, efficiency and accuracy tradeoffs remain a key limitation. In response to this gap, we introduce BlindTuner, a system that enables privacy-preserving fine-tuning by training data efficient image transformers (DEiT) directly on homomorphically encrypted data. Additionally, BlindTuner performs privacy-preserving inference, extending its usability beyond training. Experimental results show that BlindTuner achieves accuracy comparable to unencrypted models while delivering training speedups between <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.16{\times }$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$600{\times }$ </tex-math></inline-formula> over existing approaches. Furthermore, we explore its application in privacy-preserving federated fine-tuning, enabling solution for collaborative learning without compromising privacy.