Detecting And Mitigating Incoherent Input Of Latent Diffusion Models

Incoherent input is a common source of undesirable output of Latent Diffusion Models. We propose an approach for detecting and mitigating such inputs. For prompt incoherences, we analyze key categories for conflicting descriptions. We mitigate conflicts using a graph-based approach, minimizing the required modifications. Image incoherences are detected by assessing if a global or local style is present. Regions with no distinct style are discarded and replaced with a text prompt describing the image content. We evaluate the framework on over 11,000 images generated with Stable Diffusion XL using the PartiPrompts and StyleBench datasets. The framework detects 91% of all incoherences. For text input, the aesthetic score increased by up to 1.5% and the CLIP score by 4.7%. For image input, the aesthetic score improved by up to 10.1% and the CLIP score by 9.1%. In two validation surveys with 106 participants, the output of the proposed framework was preferred for 86% of the images.