Instructing Generative Image Models - From Semantics to Society

Generative image models are at the center of a rapid shift in the creative industry and the subject of an ever-growing number of research works. In this thesis, we demonstrate and address significant limitations these models suffer from despite their impressive capabilities. We highlight how naive text-to-image generation alone is insufficient for users to take charge of a model’s outputs. In particular,we showcase issues with semantic control over generated objects and scene composition and general problems like safety concerns and unintended biases. Our contributions in addressing these concerns are twofold. On the one hand, we propose new benchmarks and conduct thorough empirical studies to assess and quantify the ethical concerns in text-to-image models. Not only do our results shed light on the problematic behavior of many current models, but our methodology and insights can guide the improvement of future releases. On the other hand, we propose novel, efficient, and versatile algorithms to improve the semantic control over text-to-image models in general. Our approaches can be applied to achieve desired scene compositions or image aesthetics but also to counteract the uncovered ethical concerns. Specifically, Multifusion adds multimodal and multilingual capabilities to image generation models, SEGA allows for semantic steering of the generation process, and LEdits++ enables editing of real images. Lastly, Safe and Fair Diffusion allow users to flexibly mitigate the generation of unsafe and biased images, respectively. Importantly, our methods require comparatively little or no training and generalize to arbitrary neural network architectures.