The Cost of Convenient Validation: Subject Leakage Inflates Performance in EEG Sleep Stage Classification

Automated sleep stage classification from electroencephalography (EEG) has been widely explored using classical machine learning and deep learning models, but many studies evaluate performance using record-level rather than subject-independent splits, which can overestimate real-world generalization. This study compares Random Forest and convolutional neural network (CNN) models for five-class sleep stage classification (Wake, N1, N2, N3, REM) using three subjects from the Sleep-EDF Expanded dataset. Models were evaluated under both a conventional record-level split and a strict leave-one-subject-out (LOSO) cross-validation scheme. Using spectral and statistical EEG features, Random Forest achieved 73.7% accuracy (F1 = 0.738) under record-level validation, decreasing to 65.0% accuracy (F1 = 0.636) under LOSO validation. Feature-based CNN performance decreased from 64% to 56.4%, while the raw-signal CNN performed worst under both evaluation methods. These results demonstrate that validation strategy strongly influences conclusions about EEG model performance. In this small EEG dataset, classical machine learning with engineered features generalized more reliably across subjects than deep learning approaches. Future work will evaluate these findings using larger subject populations and additional architectures.