TINST-Net: A Novel Neural Style Transfer using Mixture of Text and Image

Traditional methods for image style transfer typically require a reference style image to learn and apply the style of it to the original image. Recently, there have been some attempts to generate new images with a specified style defined by text conditions. Both approaches have yielded impressive output images. This paper proposes a novel method for image style transfer that leverages both a style image and a text description. The text description can provide additional supervision for the style transfer process in two ways. When the text description aligns with the style of the reference image, it acts as supplementary information, potentially refining the transferred style. When it describes a different style than the reference image, the resulting image will have a blend of both styles, opening up opportunities for creative ideas. Our framework, called TINST-Net (Text and Image Neural Style Transfer), utilizes two pre-trained models for style extraction. A CLIP model extracts style information from the text description, while a VGG-19 model extracts style features from the reference image. For image generation, we use the U-Net architecture. To train the U-Net network to generate images with the target style, we introduce a new loss function. This loss function combines style loss, content loss, and CLIP loss. We experimented on a large number of images and quantitatively evaluated the quality of the generated images using Mean Opinion Score (MOS) with evaluations from 45 users. Experimental results demonstrate that TINST-Net can generate images of comparable quality, achieving a MOS of 3.5 out of 5. We release our code and experiment images at https://github.com/daitq392/TINST-Net