A Doodle-Based Control for Characters in Story Visualization

We propose a story visualization technique that allows users to control the arrangement, poses, and styles of characters in a scene based on user-input doodle sketches. Our method utilizes a text encoder to process scene prompts and an image encoder to handle doodle sketches, generating inputs for a predefined scene generation model. Furthermore, we achieve efficient model training by fine-tuning the backbone network by applying a small dataset and employing a LoRA-based fine-tuning technique. We demonstrate that our method can generate characters with various poses and styles from doodle sketches, and it can validate the advantages of our approach by comparing it with the results from other story visualization studies.