ATHEENA: Automated Toolflow for Hardware Early-Exit Network Acceleration

The continued need for improvements in accuracy, throughput, and efficiency of Deep Neural Networks has resulted in a multitude of static parameter reduction methods, like quantisation and pruning, which leverage the custom architectures possible on FPGAs. However, the potential of these solutions is already well exploited, reaching the limits of what can be achieved by reducing the number or size of the parameters while still maintaining accuracy. We propose a shift of focus to input-dependent computation to improve efficiency and reduce the average compute required for inference. Early-Exit (EE) networks have become an increasingly popular way to implement dynamic parameter reduction by varying network depth, essentially customising the computation level according to the difficulty of an input at run-time. We create Automated Toolflow for Hardware Early-Exit Network Acceleration (ATHEENA), an automated, open-source CNN-to-FPGA toolflow which utilises the probability of samples exiting early from EE networks to optimally allocate the limited resources of an FPGA to different sections of the network. This ultimately results in improved throughput. The toolflow uses the data-flow model of the existing fpgaConvNet tool, extended to support Early-Exit networks, as well as Design Space Exploration (DSE) to optimise the generated streaming architecture hardware with the goal of increasing throughput/reducing area while maintaining accuracy. To this end, we incorporate abstracted hardware models, based on Queueing Theory, to aid the DSE with a more accurate analysis of performance and resource requirements. This improves the robustness of the accelerator. Experimental results on three different networks demonstrate a throughput increase of 2.00 to 3.12 times compared to an optimised baseline network implementation with no early exits. Additionally, the toolflow can achieve a throughput matching the same baseline with as low as 48% of the resources the baseline requires.