Steering Diffusion Models via Color-Activating Noise Patches for Controllable Image Synthesis

While diffusion models have achieved remarkable progress in text-to-image (T2I) synthesis, a critical question remains: how can we achieve precise and reliable control over object colors and positions in the generated images? In this work, we first aim to answer this question by uncovering and leveraging a novel structure in the initial random Gaussian noise, which we term Color-Activating Noise (CAN) patches. These CAN patches are localized regions of the initial noise that consistently and intrinsically couple with specific colors, i.e. , they can be reliably activated by the color-related tokens in the input prompt during the denoising (generation) process. This activation, in turn, steers the position of the associated object. Based on this observation, we then propose a mining pipeline to discover CAN patches efficiently from a pre-trained UNet-based diffusion model and organize them into a plug-and-play database for retrieval and injection at inference time. We demonstrate that CAN patches successfully align the capability of UNet-based diffusion models with that of the heavily-trained Diffusion Transformers (DiTs) in reliable color-position control, particularly for atypical and multi-color generation cases, without additional optimization. We further show that CAN patches can generalize to DiT-based models, improving the generation accuracy via better object position and color control. Finally, we showcase two applications, interactive image editing and counting-aware image generation, to highlight the versatility of our CAN patches. Our method can serve as a new baseline to shed light on improving the controllability and faithfulness of diffusion-based T2I generation. Project page: https://github.com/RongjinGuo/CAN-Patches.