Multi-Modal Contextual Feature for the IP Consistent Generation

The task of IP-consistent generation aims to produce new target images of a specific IP character based on one or more reference images. With advancements in text-to-image diffusion models, this task has seen notable progress. Conceptually, it can be framed as a concept learning problem, where the unique IP concept is integrated into the image generation process. Existing methods often encode IP concepts into a textual token or inject reference image features through additional modules. However, these single-modality approaches face challenges in maintaining IP feature consistency and ensuring controllability of the generated content via textual descriptions. Consequently, they often struggle to balance IP fidelity with text-driven control. To address these limitations, we propose a novel method called Multi-Modal Contextual Feature (MMCF), which leverages multi-modal contextual information to guide the IP-consistent generation process. By introducing an adapter into the diffusion model, MMCF enhances the effective utilization of multi-modal features, enabling more accurate and harmonious control signals for the image generation process. Experimental results demonstrate that MMCF achieves better integration and interaction between textual and visual modalities, resulting in more coherent and visually consistent outputs. Compared to methods that inject control signals from the two modalities independently and in parallel, MMCF produces superior performance across various evaluation metrics. As a pioneering effort in leveraging multi-modal contextual features, this work underscores the potential of multi-modal approaches for advancing IP-consistent generation tasks.