EEG-Based Contrastive Learning Models For Object Perception Using Multisensory Image-Audio Stimuli

Multimedia sources such as images and audio commonly activate human senses to perceive objects, but limited research has explored the combined effect of these stimuli on predicting semantic object perception. In this study, we compare the performance of EEG signals elicited by image and audio stimuli in classifying semantic objects, revealing that image stimuli are more discriminative than audio stimuli. Building on this, we developed a contrastive learning model that integrates image and audio stimuli, further enhancing classification performance. Our research makes several key contributions: it compares classifier performance with uni-sensory versus multisensory stimuli, demonstrates improved performance with contrastive learning models using EEG data from both image and audio stimuli, and introduces a novel method to generate positive and negative pairs for contrastive learning models using cross-sensory EEG data. These findings enhance our understanding of how humans perceive multimedia sources and highlight the potential of multisensory integration in EEG-based classification.