Connecting Sight and Sound through Space, Time and Language

Humans perceive their environment through sight and sound, two interconnected senses that offer rich information about the world. In everyday life, we constantly experience the interaction between sounds and visuals. Various studies have explored the cross-modal associations between vision and sound, such as semantic correspondence and temporal correspondence. However, many other forms of audio-visual correspondence still have yet to be explored. In this thesis, we examine underexplored audio-visual correspondences across space, time, and language and extend beyond these aspects, demonstrating how they can be leveraged in a self-supervised manner. Specifically, we explore and tackle four problems in the audio-visual domain. (1) First, we investigate the geometric consistency between visual and spatial audio from different viewpoints. By leveraging this correspondence, we demonstrate that we can jointly learn sound localization and camera rotation estimation from unlabeled audio-visual data. (2) We explore how ambient sounds can provide information about 3D scene structure. We develop models to predict depth and obstacle locations from audio alone, showcasing the rich spatial information embedded in ambient sounds. (3) We introduce a video-guided sound generation framework by training on paired audio-video-text data to learn semantic and temporal association among three modalities. It allows users to use text input to adjust semantic associations while preserving temporal alignment for generated audio. (4) Finally, we investigate the statistical similarity between natural images and spectrograms and present the visual correspondence between them, highlighting their overlapping distributions. This allows us to generate visual spectrograms that look like images but can also be played as sound using pretrained diffusion models. Through these studies, we show how audio-visual correspondence can enhance our understanding of spatial perception, scene interpretation, and cross-modal correlation.