A novel approach for visual and semantic image reconstruction from brain activity

Significant advancements have been made in hierarchical feature decoding from human brain activity, particularly using functional magnetic resonance imaging (fMRI) data. Reconstructing images from fMRI patterns has gained considerable attention, with deep learning methods driving progress. However, challenges remain in achieving high-quality visual and semantic reconstructions. To address these limitations, a novel approach is proposed that reconstructs both visual and semantic content. For visual reconstruction, ResNet101 is employed to extract features from training images. These features, along with fMRI activity patterns, are used to train a decoder that decodes hierarchical visual features from brain data. A deep generator network produces an image, which is iteratively optimized to minimize the error until its features align with brain-decoded features. For semantic reconstruction, Bootstrapping Language-Image Pre-training (BLIP) and Latent Diffusion Model (LDM) are integrated. BLIP extracts image semantic features, which are used to train a decoder with training fMRI data. The trained decoder decodes semantic features from brain-decoded data, which are then used to generate captions. The LDM then reconstructs the final semantic image using the visually reconstructed image and extracted captions. Unlike prior studies that either relied on pre-decoded brain features or focused on a single reconstruction pathway, our framework integrates both visual and semantic reconstruction in a unified pipeline. This combined strategy demonstrates, for the first time, that state-of-the-art models such as ResNet101, BLIP, and LDM can be effectively adapted to smaller-scale fMRI datasets, yielding promising results in both visual fidelity and semantic richness. Our approach achieves high accuracy in both visual and semantic reconstructions, improving performance in quality and quantitative metrics. An AlexNet2 of 98.6% for visual reconstruction and a Contrastive Language-Image Pre-training (CLIP) distance of 0.296 for semantic reconstruction were achieved, demonstrating superior results.