Privacy-Diffusion: Privacy-Preserving Stable Diffusion Without FHE and Differential Privacy

Text-to-image generation is trending in the generative artificial intelligence (GenAI) field. Among open-sourced image generation projects, Stable Diffusion is the state-of-the-art. Many artists and service providers customize the diffusion model to generate featured high-quality images. However, there is no protection to the privacy of the input text prompt, output image, and customized model. Privacy is very important since it can increase users’ willingness to use the service and protect the service provider’s intellectual property. Existing privacy-preserving diffusion model require fully homomorphic encryption (FHE) to ensure its privacy and security. Nonetheless, FHE is very time-consuming and may reduce accuracy due to approximations and deteriorate image quality. In this research, we propose Privacy-Diffusion, a privacy-preserving diffusion framework without FHE. By utilizing the irreversible property of neural network layers and the property that the predicted noise in the diffusion process is a normalized Gaussian distribution. Our framework can be applied to all kinds of diffusion models to protect clients’ input text prompt and the generated image from being learned by the server, as well as customized models from being learned by the clients. Our protocol is secure and efficient. Compared with existing research, HE-diffusion, which spent 200% extra time and visible quality loss, our protocol can reach the same security level with only 19% extra time and has no quality loss. To the best of our knowledge, our Privacy-Diffusion is the first protocol that achieves this goal without using FHE and maintain the same high-quality image output as the original model.