EEG-Conformer for EEG Decoding: A Participant-Wise Evaluation and Ablation Study Across Three Datasets

Abstract Hybrid convolutional–Transformer networks are increasingly used for electroencephalography (EEG) decoding, but their components' contribution and the stability of reported effects are rarely tested. We evaluated the published EEG-Conformer against EEGNet, ShallowConvNet and DeepConvNet on a custom dataset of 48 healthy participants (eyes-open, eyes-closed, hands-up; 19 EEG and 3 accelerometer channels at 500 Hz) and on subsets of PhysioNet and BCI Competition IV-2a, using participant-wise splits. In a single split, EEG-Conformer reached 67.27% three-class accuracy, close to EEGNet (67.00%), and removing the accelerometer channels appeared to raise accuracy by 2.9 points. Five repeats of five-fold cross-validation with different partitions and seeds reversed the accelerometer effect: removing the accelerometer channels lowered accuracy by 1.2 points (95% CI 0.1–2.3) and removing the Transformer by 0.7 points (0.2–1.3), both small against a between-participant SD of 8–9 points. On PhysioNet, the single-split Transformer benefit (2.1 points) disappeared in repeated cross-validation (− 0.2 points; 95% CI − 1.4 to 1.0). BCI IV-2a leave-one-subject-out validation showed no clear benefit (+ 0.5 points; 95% CI − 1.5 to 2.5). Exploratory tests found no clear loss after downsampling to 125 Hz or restricting the band to 4–13 Hz, whereas keeping only 13–40 Hz cost 5.6 points. Hands-up recall (about 50%) showed no detectable change under band or channel restrictions, with wide intervals. The fixed recording order remains a confound that a within-recording probe could not resolve. Trial-level splitting yielded 13.4-point higher BCI IV-2a accuracy, although training size and test composition also differed; single-split estimates can misstate even the sign of small effects.