Load Balanced Sparse Bundle Adjustment Accelerator for CNN-Based Visual-Inertial Odometry on FPGA

Visual-Inertial Odometry (VIO) is a fundamental module in unmanned systems, and Convolutional Neural Network (CNN)-based feature extraction methods have been increasingly adopted in recent years. As Domain-Specific Architectures (DSA) for CNN have advanced, Bundle Adjustment (BA) in the VIO backend, rather than CNN, has become the computational bottleneck in embedded systems. However, the CNN accelerator consumes substantial FPGA resources, leaving limited capacity for BA. To address this issue, we design an efficient BA accelerator that utilizes the sparsity of the data and employs a workload segmentation and subtask scheduling architecture. Additionally, we propose a method to compress the frontend feature extraction algorithm. Experiments on the ZCU102 and a small drone demonstrate that our BA accelerator achieves <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$30.3 \times$</tex> speedup over the ARM CPU, and higher resource efficiency compared to recent accelerators. The frontend and backend of the VIO system are accelerated by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3.1 \times$</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$3.7 \times$</tex>, respectively, resulting in an overall processing speed of 19.7 fps in real world.