Multi-modal deep learning architecture for enhanced feature extraction and classification of imagined speech words

Abstract Imagined speech prediction is a challenging task with significant implications for brain-computer interfaces (BCIs) and assistive communication technologies. EEG signals are highly non-stationary and thus it is extremely difficult to find any relevant information from these electroencephalogram (EEG) signals just by seeing them in time domain. Imagined speech-based BCIs can be integrated into assistive technology devices that are specifically designed for individuals with severe physical disabilities and can also play a role in rehabilitation programs for individuals recovering from conditions that affect speech and mobility. This paper proposes a novel hybrid model that combines convolutional neural network (CNN) and recurrent neural networks (RNNs) to predict imagined speech patterns from electroencephalography (EEG) signals. The integration of CNNs allows the model to capture spatial information from multi-channel EEG data, while RNNs excel at capturing temporal dependencies inherent in sequential EEG signals. The proposed model architecture starts with a 1-D CNN layer to extract spatial features from raw EEG data. The output is then fed into RNNs to capture temporal patterns and relationships over time. It achieved competitive accuracy, averaging approximately 85.25, 87.50, 81.58 and 93.92 percent for short words, long words, vowels and short-long words respectively.