Real-time small object detection on embedded hardware for 360-degree vision

Recently deep learning-based methods for small object detection have been improved by leveraging temporal information. The capability of detecting objects down to five pixels, provides new opportunities for automated surveillance with high resolution wide field of view cameras. However, integration on unmanned vehicles generally comes with strict demands on size, weight and power. This poses a challenge for processing high framerate high resolution data, especially when multiple camera streams need to be analyzed in parallel for 360 degrees situational awareness. This paper presents results of the Penta Mantis-Vision project where we investigated the parallel processing of four 4K camera video streams with commercially available edge computing hardware, specifically the Nvidia Jetson AGX Orin. As the computational power of the GPU on an embedded platform is a critical bottleneck we explore widely available techniques to accelerate inference or reduce power consumption. Specifically we analyze the effect of INT8 quantization and replacement of the activation function on small object detection. Furthermore we propose a prioritized a tiling strategy to process camera frames in such a way that new objects can be detected anywhere in the camera view while previously detected objects can still be tracked robustly. We implemented a video processing pipeline for different temporal YOLOv8 models and evaluated these with respect to object detection accuracy and throughput. Our results demonstrate that recently developed deep learning models can be deployed on embedded devices for real-time multi-cam detection and tracking of small objects without compromising object detection accuracy.