Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment

How does audio describe the world around us? In this work, we propose a method for generating images of visual scenes from diverse in-the-wild sounds. This cross-modal generation task is challenging due to the significant information gap between auditory and visual signals. We address this challenge by designing a model that aligns audio-visual modalities by enriching audio features with visual information and translating them into the visual latent space. These features are then fed into the pre-trained image generator to produce images. To enhance image quality, we use sound source localization to select audio-visual pairs with strong cross-modal correlations. Our method achieves substantially better results on the VEGAS and VGGSound datasets compared to previous work and demonstrates control over the generation process through simple manipulations to the input waveform or latent space. Furthermore, we analyze the geometric properties of the learned embedding space and demonstrate that our learning approach effectively aligns audio-visual signals for cross-modal generation. Based on this analysis, we show that our method is agnostic to specific design choices, showing its generalizability by integrating various model architectures and different types of audio-visual data.

Authors

Publication Details

Journal
International Journal of Computer Vision
Published
2026-11-21
DOI
https://doi.org/10.1007/s11263-026-03106-5
Citations
7
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
3.17
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment

Arda Senocak, Sung-Bin Kim, Tae-Hyun Oh, Hyunwoo Ha
7 citations
International Journal of Computer Vision
Multimodal Machine Learning Applications
3.17
article

Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment

Arda Senocak, Sung-Bin Kim, Tae-Hyun Oh, Hyunwoo Ha
article en
7 citations

Abstract

How does audio describe the world around us? In this work, we propose a method for generating images of visual scenes from diverse in-the-wild sounds. This cross-modal generation task is challenging due to the significant information gap between auditory and visual signals. We address this challenge by designing a model that aligns audio-visual modalities by enriching audio features with visual information and translating them into the visual latent space. These features are then fed into the pre-trained image generator to produce images. To enhance image quality, we use sound source localization to select audio-visual pairs with strong cross-modal correlations. Our method achieves substantially better results on the VEGAS and VGGSound datasets compared to previous work and demonstrates control over the generation process through simple manipulations to the input waveform or latent space. Furthermore, we analyze the geometric properties of the learned embedding space and demonstrate that our learning approach effectively aligns audio-visual signals for cross-modal generation. Based on this analysis, we show that our method is agnostic to specific design choices, showing its generalizability by integrating various model architectures and different types of audio-visual data.

International Journal of Computer VisionVol. 134(12)
Openalex Percentile: Top 9%
Multimodal Machine Learning Applications
3.17
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment — Arda Senocak, Sung-Bin Kim, et al. · International Journal of Computer Vision (2026) | TGRS Research Map | TGRS