Towards neural foundation models for vision: Aligning EEG, MEG, and fMRI representations for decoding, encoding, and modality conversion

This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across cross-modal representations of brain activity by leveraging contrastive learning. We leverage electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. The framework is validated through three key experiments: decoding visual information from neural data, encoding images into neural representations, and converting between neural modalities. The results highlight the model’s ability to accurately capture semantic information across different brain imaging techniques, underscoring its potential in decoding, encoding, and modality conversion tasks. Our results reveal that EEG, MEG, and fMRI signals—despite being collected from different subjects and datasets—can be aligned in a shared vision-grounded space that preserves semantic information. This finding suggests the existence of modality-invariant neural codes and offers a new framework for comparing and decoding brain activity across heterogeneous non-invasive recordings. • Unified Multimodal Representation : Aligns EEG, MEG, and fMRI data with visual stimuli using contrastive learning for a shared representation. • Three Key Capabilities : Enables neural data decoding, visual encoding, and cross-modality conversion, showcasing versatility. • High Decoding Accuracy : Achieves 93% CLIP 2-way accuracy with fMRI, matching or surpassing similar models. • Cross-Modality Consistency: Translates semantic content effectively across EEG, MEG, and fMRI. • Towards a Neural Foundation Model : Advances a unified framework for neural data, similar to NLP foundation models.