An Energy-Efficient Neuromorphic Accelerator Based on Deformable Spiking Transformer for Dynamic Vision Sensor

Nowadays, brain-inspired Spiking Neural Networks (SNNs) have been proven to effectively process Dynamic Vision Sensor (DVS) event data streams due to their event-driven computation and temporal characteristics. Among the various SNN models, the spiking-based Transformer has demonstrated superior performance. However, there is few effort focused on designing dedicated accelerators for spiking-based Transformers. In this paper, we propose an energy-efficient neuromorphic accelerator based on a novel spiking-based Transformer for DVS. At the algorithmic level, we propose a Deformable Spiking Transformer (DST), which incorporates novel Spike-Driven Deformable Attention modules to enhance feature extraction while reducing computational complexity. At the hardware level, we design an energy-efficient Spiking Convolution Core and Spiking Attention Core to efficiently support sparse spiking convolutions and deformable attention mechanisms in the DST. Moreover, to leverage dynamic sparsity and minimize processing latency, we introduce a sparse spiking computing flow that enables parallel processing of sparse computations in the spiking convolutions. Based on algorithm-hardware co-optimization, we develop an energy-efficient neuromorphic accelerator for DVS processing and implement it in TSMC 28nm CMOS technology. Experimental results show that the DST achieves promising accuracy on the DVS datasets while maintaining competitive inference latency. Compared to prior hardware designs for SNNs, the proposed accelerator has the highest peak throughput. In comparison to Transformer accelerators, it achieves at least <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.11 \times $</tex-math> </inline-formula> improvement in energy efficiency.