Application of multimodal fusion generative adversarial networks in cross-domain artistic style conversion

Abstract Cross-domain artistic style conversion is a challenging task that requires integrating multiple modalities. Leveraging advanced Generative Adversarial Networks (GAN) architectures, this research aims to improve cross-domain style transfer by integrating multimodal fusion techniques. The objective is to develop an Intelligent Whale Optimized Time Series GAN (IWO-TSGAN) model for high-quality cross-domain artistic style conversion. The model aims to merge different modalities effectively to achieve accurate and coherent artistic transformations while preserving content and style consistency. Multimodal data was collected, comprising high-resolution images of artwork styles, textual descriptions, and audio narrations. Image preprocessing involved resizing for consistency. Text data underwent tokenization to convert descriptions into structured formats for input. Audio data was normalized to ensure uniformity across samples, enabling better fusion across different modalities for style conversion. Feature-level fusion is performed at the intermediate layer level, where extracted features from image, text, and audio modalities are combined. A novel technique inspired by the IWO Algorithm was used to fine-tune the GAN’s performance in cross-domain artistic style conversion. Using Python, the model achieved impressive results in cross-domain style conversion. Metrics such as SSIM (0.890), FID, and PSNR above 25dB, were used to evaluate image quality, showing a significant improvement over traditional models in handling multimodal fusion. The proposed IWO-TSGAN model successfully enhances cross-domain artistic style conversion by integrating multimodal data sources. The results demonstrate significant improvements in transfer quality, offering promising applications in digital art creation, multimedia content generation, and interactive media.