Optimization of DNN-based HSI Segmentation FPGA-based SoC for ADS: A Practical Approach

The use of hyperspectral imaging (HSI) for autonomous navigation is a promising field of research that aims at improving the accuracy and robustness of detection, tracking, and scene understanding systems based on vision sensors. The combination of advanced computer algorithms, such as deep neural networks (DNNs), and small-size snapshot HSI cameras allows to strengthen the reliability of those vision systems. Using HSI, some intrinsic limitations of greyscale and RGB imaging in depicting physical properties of targets related to the spectral reflectance of materials (metamerism) are overcome. Despite the promising results of many published HSI-based computer vision developments, the strict requirements of safety-critical applications such as autonomous driving systems (ADS) regarding latency, resource consumption, and security are prompting the migration of machine learning (ML)-based solutions to edge platforms. This involves a thorough software/hardware co-design scheme to distribute and optimize the tasks efficiently among the limited resources of computing platforms. With respect to inference, the over-parameterized nature of DNNs poses significant computational challenges for real-time on-the-edge deployment. In addition, the intensive data preprocessing required by HSI, which is frequently overlooked, must be carefully managed in terms of memory arrangement and inter-task communication to enable an efficient integrated pipeline design on a system on chip (SoC). This work presents a set of optimization techniques for the practical co-design of a DNN-based HSI segmentation processor deployed on a field programmable gate array (FPGA)-based SoC targeted at ADS, including key optimizations such as functional software/hardware task distribution, hardware-aware preprocessing, ML model compression, and a complete pipelined deployment. Applied compression techniques significantly reduce the complexity of the designed DNN to 24.34% of the original operations and to 1.02% of the original number of parameters, achieving a 2.86× speed-up in the inference task without noticeable degradation of the segmentation accuracy.