Video-based epileptic seizure classification: A novel multi-stage approach integrating vision and motion transformer deep learning models

Automated seizure quantification and classification are needed for semiology-based epileptic seizure diagnosis support. To the best of our knowledge, the 5-class (Hypermotor, Automotor, Complex Motor, Psychogenic Non-Epileptic Seizures, and Generalized Tonic-Clonic Seizures) seizure video dataset (198 seizures from 74 patients) studied in this paper is the largest 5-class dataset ever curated, composed of monocular RGB videos from two university hospital epilepsy monitoring units. 2D skeletons were estimated using ViTPose, a vision transformer deep learning (DL) architecture, and lifted to 3D space using MotionBERT, a multimodal motion transformer architecture. The movements were quantified based on the estimated 3D skeleton sequences. Two approaches were evaluated for seizure classification: (1) classical machine learning methods (Random Forest (RF) and XGBoost) applied to quantified movement parameters, and (2) 2D skeleton-based DL using MotionBERT action, an action recognition DL model, to which we perform transfer-learning. The best model achieved a promising, above literature, 5-fold cross-validated macro average F1-score of 0.84 ± 0.09 (RF) for 5-class classification. The binary case (Automotor vs Hypermotor) resulted in 0.80 ± 0.18 (MotionBERT action), and adding a 3 rd class (Complex motor) lowered to 0.65 ± 0.14 (RF). This novel multi-stage classification ensures that the included movement features are traceable, allowing interpretable AI exploration of this novel approach supporting future clinical diagnosis. • Multi-stage classification ensures traceability and interpretable AI. • Largest 5-class seizure dataset: Hypermotor, Automotor, Complex Motor, PNES, GTCS. • 2D and 3D skeletons estimated using pre-trained ViTPose and MotionBERT. • Fully automatic motion quantification based on RGB video footage. • Achieved a 5-class validation F1-score of 0.84 ±0.09 using Random Forest.