ContextDiff: Context-aware controllable generation with text-to-image diffusion models

Recent advancements in image generation have explored in-context learning by presenting a query image alongside context examples and text prompts. Despite these efforts, the generated images often exhibit diminished quality and fidelity when text prompts are omitted, highlighting a significant limitation in the models’ ability to fully exploit visual context. To address this, we introduce Context-Aware Controllable Generation (ContextDiff), a pioneering task designed to generate semantically precise and contextually rich images within specified categories. Our approach establishes a context space and develops a mapping network that aligns this space with the powerful Contrastive Language-Image Pre-training (CLIP) framework, thereby providing a concrete representation of abstract context. To ensure semantic diversity and contextual fidelity, we propose three novel loss functions. Extensive experiments and user studies demonstrate that our model significantly outperforms existing models, resulting in marked improvements in image quality and fidelity.