A modular approach to enhancing safety in text-to-image diffusion models via targeted LoRA fine-tuning

This paper discusses recent advances in text-to-image generative models, which improve image quality and prompt alignment but also introduce vulnerabilities to harmful content from adversarial prompts.Existing safety measures often fail against complex attacks.We propose a novel safety approach using targeted Low-Rank Adaptation (LoRA) fine-tuning combined with Metric learning.Our approach aimed for modification of latent vectors of harmful prompts, aligning them with safe content to reduce unsafe outputs.Experiments demonstrate safety levels comparable to industry benchmarks, particularly when adapters are trained on specific harm categories.Our study provides a reusable framework to protect against harmful outputs that can be scaled to protect against multiple prompt categories.