Energy Efficient 3D CNN Inference Using Multi-Dimensional Systolic Architectures on FPGA

The emergence of 3D Convolutional Neural Networks (CNNs) has revolutionized video-related tasks by enabling efficient processing of spatiotemporal data. In this paper, we have proposed a new hardware design for efficient mapping of 3D CNN on 3D systolic array architecture. The architecture enables data reuse across both spatial and temporal dimensions to enhance computational efficiency. Our architecture reduces memory utilization by eliminating the need to store large, replicated datasets typically required for convolution operations. Moreover, we have introduced a generalized dataflow model tailored for 3D systolic architectures to further increase the throughput. The designed hardware accelerator leverages 3D dataflow to maximize hardware utilization and minimize data transfer latency. An FPGA based implementation of a pretrained 3D CNN for human activity recognition was carried out. The efficacy of the proposed approach in terms of energy efficiency and latency was evaluated for widely used 3D CNNs including C3D, I3D, and R(2+1)D. The results show that our design minimizes the latency and energy dissipation by around 50%.