Reverse the auditory processing pathway: Coarse-to-fine audio reconstruction from human brain activity

Drawing inspiration from the hierarchical pro-cessing of the human auditory system, which transforms sound from low-level acoustic features to high-level semantic understanding, we introduce a novel coarse-to-fine audio reconstruction method. Leveraging non-invasive functional Magnetic Resonance Imaging (fMRI) data, our approach first utilizes CLAP to decode fMRI signals coarsely into a se-mantic space, followed by a semantically-guided fine-grained decoding into the AudioMAE latent space. These fine-grained neural features then serve as conditions for high-fidelity audio reconstruction through a Latent Diffusion Model (LDM). Ex-tensive validation on three public fMRI datasets demonstrates the superiority of our coarse-to-fine decoding method over conventional fine-grained approaches, achieving state-of-the-art performance across metrics including FD, FAD, and KL. Furthermore, reconstruction quality in challenging scenar-ios is enhanced through an innovative semantic prompting mechanism. This framework holds potential for advancing brain-computer interfaces and assistive technologies, such as improved hearing aids and neural communication systems for those with auditory or speech impairments. Reconstructed results are available at <a href="https://neurofusex.github.io/c2f-ldm/">https://neurofusex.github.io/c2f-ldm/</a><strong>.</strong>