Multimodal Time-Series Recognition With Spatio–Temporal Dynamic Graph Spike Neural Networks

Brain–computer interface (BCI) is a cutting-edge technology that holds promise in the healthcare industry, where multimodal information integration is required to capture features of physiological signals from different equipments. Challenge remains as how to integrate and detect diversified time series, such as electroencephalogram (EEG) and audio, for anomaly detection. This article proposes a spatio–temporal dynamic graph spike neural network, termed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DynGraphSpike</i>, which is constituted by two brain-inspired perception modules, each driven by a dynamic graph neural network (DGNN), and one spike-based multisensory integration network. DGNN incorporates the Wilson-Cowan model to configure spatio– temporal dynamics of the nodes, and utilizes the phase locking values of EEG channels to reshape the edges. To merge EEG and audio, spiking neurons are used to construct the multisensory integration network. Experiments demonstrate that DynGraphSpike achieves a classification accuracy over 99%, outperforming the state-of-the-art methods. Ablation studies confirm that the Wilson–Cowan and spiking dynamics enhance feature alignment via temporal synchronization, as indicated by dynamic warping time scores. Together with spatio–temporal dynamics, DynGraphSpike has the potential to facilitate BCI technologies for disease diagnosis.