High Accuracy Is Not Generalization: Evaluation Design in EEG Mental Arithmetic Classification
Classification of rest versus mental arithmetic in the public EEGMAT dataset is often reported with accuracies of 88–99%. Such results usually come from subject-dependent designs, in which epochs from the same person appear in both training and test sets. This design is often intentional and can be appropriate for within-subject prediction. It becomes misleading, however, when the resulting accuracy is read as evidence that a model will work on unseen subjects. We tested how much of the reported accuracy survives a stricter evaluation. Balancing and augmentation were confined to the training data, test sets kept their natural class ratio, and models were selected on validation data. A multilayer perceptron on 57 simple time-domain features was evaluated under four protocols, with over 1000 trained networks and subject-level confidence intervals. Balancing before splitting gave kappa 0.77–0.88, with 45–47% of test items duplicated in training. A correct random epoch split gave kappa 0.29–0.58, so duplicate contamination explains most of the inflation. Leave-one-subject-out evaluation gave kappa 0.19–0.26 and ROC-AUC 0.66–0.71, which is above chance but modest. The remaining subject-dependent advantage (kappa difference 0.06–0.32) is consistent with subject-identity leakage but does not prove it. To our knowledge, no previous study on EEGMAT has examined this question. Results refer to the available 36-subject pool, not to the general population.
Authors
- Monica Leba (ORCID: https://orcid.org/0000-0002-4083-9121)
- Andreea Cristina Ionica (ORCID: https://orcid.org/0000-0003-1988-9340)
- Remus Sibisanu
Institutions
- Universitatea Din Petrosani (RO)
Publication Details
- Journal
- Entropy
- Published
- 2026-11-19
- DOI
- https://doi.org/10.3390/e28111275
- Primary Topic
- EEG and Brain-Computer Interfaces
- Type
- article
- Field-Weighted Citation Impact
- 0.00