Systematic Ablation Study of Encoder Only Transformers for Epileptic Seizure Monitoring

The precise identification of epileptic stage from electroencephalogram (EEG) signals holds substantial importance for the diagnosis, monitoring, and prediction of epileptic seizures which in turn can significantly contribute to patient care and quality of life. Transformer architectures have demonstrated noteworthy potential across a spectrum of sequence modelling tasks. However, optimizing transformer-based networks for identifying epileptic states from raw EEG requires a deeper understanding of the influence of key architectural components and data pre-processing modalities. This paper delineates a comprehensive ablation study that probes into explaining the effect of input characteristics, embedding methods, attention mechanisms and finer architectural decisions on the performance of a simple encoder-only transformer architecture from a total of 826 experiments conducted during the study. The optimal model from study achieved an average sensitivity, specificity and macro F1 score of 0.9271, 0.9635, and 0.927 respectively on the 10-fold cross-validation assessment, effectively surpassing the current benchmark for 3-class epileptic EEG classification on the CHB-MIT dataset. The study reveals answers to some of the most important decisions that need to be made when training transformers-based neural network architectures for EEG applications.