Latent Orthogonal Perturbation: A Data Poisoning Approach to Prevent Unauthorized LoRA Fine-Tuning

Generative AI (GenAI) enables creators to produce high-quality images without advanced artistic skills, but also raises concerns over the unauthorized replication of distinctive styles. With the rise of fine-tuning techniques like LoRA, mimicking specific artists’ styles has become increasingly accessible. To address this, we propose Latent Orthogonal Perturbation (LO-Perturbation), a data poisoning method that subtly injects orthogonal noise into the latent space of artworks. These perturbations introduce only subtle visual changes but significantly degrade the model’s ability to learn fine-grained stylistic features during LoRA fine-tuning. We evaluate our method using five standard image quality metrics. Results show that, while Glaze and Nightshade prioritize preserving visual fidelity, LO-Perturbation leads to significantly greater degradation in image quality and alignment under LoRA fine-tuning—achieving a 56.92% lower Q-Align score and a 46.18% higher FID, along with the lowest CLIP-IQA and LIQE scores. Our method directly disrupts the fine-tuning process, a threat that is less emphasized in existing protection techniques, thus offering an additional layer of protection for artists.