Enhancing Multimodal Zero-Calibration RSVP-Based Target Detection With Cross-Subject Mixing and Cross-Modal Alignment

Rapid Serial Visual Presentation (RSVP)-based Brain-Computer Interface (BCI) can efficiently detect rare target images by capturing event-related potentials (ERPs) evoked by target images in electroencephalogram (EEG) signals. RSVP-based BCI systems require subject-specific data collection to train the decoding model for one new user. Their performances depend on the amount of each new subject’s data for training a model, which limits practical application. Thus, it is necessary to develop the zero-calibration method in RSVP-based BCIs, requiring only historical collected data to train models for a new user. In this work, we introduce the pupil modality and develop a multi-modal method combining the EEG and pupil signals to enhance zero-calibration RSVP-based target detection performance. The proposed ResVPNet, based on the residual network tailored for RSVP signal decoding, extracts signal features for each modality with a two-branch structure. A MixSub strategy is designed to implicitly generate new signal samples by mixing subject-related information between subjects to enhance the diversity of training distributions. Also, a Prototype-based Cross-Modal Alignment (PCMA) module is employed to align multi-modal signal features and separate different class samples. Our proposed network achieved a balanced accuracy of 91.89%, surpassing the performance of the compared methods. The ablation studies and visualizations revealed the effectiveness of the proposed modules. This work provides an effective method for the zero-calibration RSVP-based target detection system.