Efficient and Hardware-Friendly Online Adaptation for Deep Stereo Depth Estimation on Embedded Robots

Accurate and real-time stereo depth estimation is important for autonomous robots, such as autonomous aerial vehicles (AAVs). Due to the computation constraints of these miniaturized robots, current state-of-the-art algorithms deploy light-weight neural networks while using self-supervised online adaptation to compensate for the lack of generalization. However, the traditional online training approach introduces 2× extra computation overhead, resulting in the failure to meet real-time requirements. Existing efficient training algorithms are primarily designed for train-from-scratch scenarios rather than online training, and involve complicated data quantization methods and non-standard operations, making them highly unfriendly to deployment on robots equipped with embedded GPUs or neural processing units (NPUs). Therefore, this paper aims to improve the online adaptation for deep stereo at the system level from both hardware and software aspects, and proposes a novel online adaptation method, which is robust, computationally efficient, and hardware-friendly. First, we adopt 8-bit quantized training strategy to maximize the performance of typical embedded computing platforms. Considering the streaming input of data during deployment, we design an online calibration method for quantized self-adaptive deep stereo. Then we only update the bias of the convolutional layers and design a plug-in layer with negligible computational cost to enhance the adaptation effect. Meanwhile, this layer is inherently compatible with existing GPUs and NPUs. Our final deep stereo system speeds up the inference and adaptation by 2.11×, which can process 640×360 resolution images at 11.1 FPS on the NVIDIA Jetson Orin NX, and obtains estimation accuracy comparable to current adaptation methods. When deployed on the Horizon Journey-5 Chip, it can further achieve a 10× speedup than Orin NX.