RETA-AD: A Reconfigurable and Efficient Transformer Accelerator for Autonomous Driving

The Transformer model is widely used in autonomous driving (AD) networks. However, diverse attention mechanisms and varying operators in AD networks lead to high computational complexity and substantial memory access requirements, presenting challenges to existing Transformer accelerators. To address these issues, we propose RETA-AD, a reconfigurable and efficient Transformer accelerator tailored for speeding up both normal attention (NA) and deformable attention (DA) computations within AD networks. First, a highly flexible 3-D Folded Dimension Systolic Array (FDSA) is developed, which is capable of processing small-width matrix multiplications (SWMMs) across various DA configurations, significantly improving hardware utilization and speed. Second, the computation of noncomputation-intensive (NCI) operators in AD networks is optimized, including a Reorganized Feature Map (RFM) sampling strategy to reduce the sampling time, and a pipeline reconfigurable (PR) Softmax module incorporating both coarse and fine-grained pipelines to support varying attention configurations with constantly high efficiency. Lastly, an Inter-tile Cross Parallel (ITCP) dataflow is designed to minimize on-chip storage requirements and hide the latency of NCI operations. The proposed RETA-AD is implemented on a Xilinx UltraScale+ FPGA development board. Experimental results show a 6.3–<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$8.9\times $</tex-math> </inline-formula> speedup and a 13.7–<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$22.5\times $</tex-math> </inline-formula> improvement in energy efficiency over the GPU A100 and the edge GPU Jetson AGX Xavier. Compared to previous FPGA-based Transformer accelerators, RETA-AD demonstrates a 1.1–<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.62\times $</tex-math> </inline-formula> speedup and a 2.20–<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.62\times $</tex-math> </inline-formula> improvement in energy efficiency.