Towards High-Performance Flexible FPGA-Based Accelerators for CNN Inference: General Hardware Architecture and End-to-End Deploying Toolflow

FPGAs are widely used for efficient CNN inference acceleration but designing high-performance accelerators demands significant hardware expertise. Existing solutions face limitations: hardware designs are often model/chip-specific with suboptimal resource efficiency, and compiler support is typically framework-restricted. To overcome these, we propose a generalized and flexible high-performance FPGA accelerator architecture and a flexible end-to-end compilation toolflow based on ONNX IR. The architecture features an optimized uint8 systolic array for high compute density and a dedicated X-bus module handling diverse convolution parameters. On-chip buffers and allocation algorithms enhance memory efficiency. Configurable design variables enable architectural adaptation and fine-tuning. Deploying four accelerator variants on a VCU118 board and compiling 17 CNN models demonstrated a peak convolutional throughput of 5792.19 GOPS (99.82% of theoretical peak, 5825.42 GOPS) and overall throughput up to 3311.48 GOPS. Compared to prior work, our solution offers superior usability, greater flexibility, and higher performance under comparable DSP usage. Furthermore, across most tested models, it provides significantly lower latency and higher energy efficiency versus CPUs and GPUs.