SGSAFormer: Spike Gated Self-Attention Transformer and Temporal Attention

Spiking neural networks (SNNs), a neural network model structure inspired by the human brain, have emerged as a more energy-efficient deep learning paradigm due to their unique spike-based transmission and event-driven characteristics. Combining SNNs with the Transformer model significantly enhances SNNs’ performance while maintaining good energy efficiency. The gating mechanism, which dynamically adjusts input data and controls information flow, plays an important role in artificial neural networks (ANNs). Here, we introduce this gating mechanism into SNNs and propose a novel spike Transformer model, called SGSAFormer, based on the Spikformer network architecture. We introduce the Spike Gated Linear Unit (SGLU) module to improve the Multi-layer perceptron (MLP) module in SNNs by adding a gating mechanism to enhance the model’s expressive power. We also incorporate Spike Gated Self-Attention (SGSA) to strengthen the network’s attention mechanism, improving its ability to capture temporal information and dynamic processing. Additionally, we propose a Temporal Attention (TA) module, which selects new filters for the input data along the temporal dimension and can substantially reduce energy consumption with only a slight decrease in accuracy. To validate the effectiveness of our approach, we conducted extensive experiments on several neuromorphic datasets. Our model outperforms other state-of-the-art models in terms of performance.