Deep Hybrid Synesthesia Model for Audio-Image Transfer

Most artistic expressions are conveyed through images (e.g., painting) and audio (e.g., music), and deep learning has been successfully applied to neural style transfer within each of these modalities. However, there is still a lack of deep models that explicitly learn to transfer style between images and audio. Motivated by synesthesia, which reflects intrinsic connections between vision and hearing in the human brain, we propose a deep hybrid synesthesia model for audio–image style transfer. Our framework consists of two main components: (1) a component conversion module that learns cross-modal mappings between audio rhythm/spectrum and image color/shape in a continuous valence–arousal (VA) emotion space; and (2) a style conversion module that transfers high-level artistic styles between Eastern (ink-wash, shui-mo) and Western painting and their corresponding musical counterparts. We first learn emotion-aware feature networks that align low-level audio and visual components based on shared affective representations, and then model long-term stylistic structures for cross-modal style transfer. Experiments include “seeing the sound” (audio-to-image generation with controllable components) and full audio–image style transformations. Both objective analyses and subjective evaluations suggest that our model can produce cross-modal artworks whose perceived style and emotional content are consistent with human synesthetic impressions.