Temporal attention fusion network with custom loss function for EEG–fNIRS classification

<i>Objective.</i>Methods that can detect brain activities accurately are crucial owing to the increasing prevalence of neurological disorders. In this context, a combination of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) offers a powerful approach to understanding normal and pathological brain functions, thereby overcoming the limitations of each modality, such as susceptibility to artifacts of EEG and limited temporal resolution of fNIRS. However, challenges such as class imbalance and inter-class variability within multisubject data hinder their full potential.<i>Approach.</i>To address this issue, we propose a novel temporal attention fusion network (TAFN) with a custom loss function. The TAFN model incorporates attention mechanisms to its long short-term memory and temporal convolutional layers to accurately capture spatial and temporal dependencies in the EEG-fNIRS data. The custom loss function combines class weights and asymmetric loss terms to ensure the precise classification of cognitive and motor intentions, along with addressing class imbalance issues.<i>Main results.</i>Rigorous testing demonstrated the exceptional cross-subject accuracy of the TAFN, exceeding 99% for cognitive tasks and 97% for motor imagery (MI) tasks. Additionally, the ability of the model to detect subtle differences in epilepsy was analyzed using scalp topography in MI tasks.<i>Significance.</i>This study presents a technique that outperforms traditional methods for detecting high-precision brain activity with subtle differences in the associated patterns. This makes it a promising tool for applications such as epilepsy and seizure detection, in which discerning subtle pattern differences is of paramount importance.