An Efficient Mixture of LoRA Experts for Instruction-based Multi-task Image Editing

This study proposes MOLIE, a novel instruction-based diffusion model for domain-specific task image editing that integrates a mixture of LoRA experts. To enhance efficiency and scalability, MOLIE combines the LoRA fine-tuning scheme with a Mixture of Experts (MoE) architecture, where individual LoRA modules are specialized trained for image editing operations such as object addition, removal, replacement, object change, cartoonization, and image deblurring. During fine-tuning, each LoRA expert module sets up its parameters for only the U-Net up blocks, reducing computational cost and enhancing image quality. A novel loss function incorporating structural loss is employed to train the gating function, enabling effective coordination of tasks among multiple LoRA experts. Model performance is evaluated using a novel Instruction-Image Alignment (IIA) metric, derived from prompts to a multi-modal large language model. Extensive quantitative and qualitative experiments on a benchmark dataset demonstrate the superior performance of the proposed model.