Efficient Tensor-Based Fine-Tuning for Subject-Driven Image Editing on Diffusion Models

With advancements of diffusion models in generating high-quality, diverse, and creative content, subject-driven image editing based on diffusion models has been growing attention. However, it is challenging to perform precise control in image editing as minor changes in the input can lead to significant differences in generated concepts and content. Prevalent methods face significant limitations in terms of large numbers of trainable parameters and inference latency. Hence, we propose an efficient tensor-based fine-tuning method, Tucker/TT-based Fine-Tuning (TuckerFT and TTFT), for subject-driven image editing on diffusion models. Specifically, we first provide Tucker/TT-based Fine-tuning Convolutional Adapter and Attention Adapter to achieve parameter-efficient fine-tuning in convolutional and attention layers of diffusion models. Additionally, we further present a more sound initializing strategy for the Tucker/TT-based Adapters to guarantee steady training. Extensive experimental results demonstrate that TuckerFT/TTFT surpasses state-of-the-art methods on subject similarity, trainable parameters, and training duration.