FineControlNet: Fine-level Text Control for Image Generation with Spatially Aligned Text Control Injection

Recently introduced ControlNet has the ability to steer the text-driven image generation process with geometric input such as 2D human pose, or edge representations. While ControlNet provides control over the geometric form of the instances in the generated image, it lacks the capability to dictate the visual appearance of each instance. We present FineControlNet to provide fine control over each instance's appearance while maintaining the pose control capability. Specifically, we develop and demonstrate FineControlNet with geometric control via human pose images and appearance control via instance-level text prompts. The spatial alignment of 2D poses and instance-specific text prompts in latent space enables the fine control of multiple instances. We evaluate the performance of FineControlNet with rigorous comparison against state-of-the-art pose-conditioned text-to-image diffusion models. FineControlNet achieves superior performance in generating high quality images that follow instance-specific controls. We will release the code and the dataset.