Agriculture surrounding monitoring and object identification based on optimized you only look once and single shot multibox detector setups using combined vision and thermal images

- This paper presents a monitoring and object identification method in agricultural environments using both vision and thermal images. We evaluate two distinct approaches: a dual-network architecture, where separate models are trained for each image, and a unified network that integrates both data types into a single processing stream. Multiple prototypes based on You Only Look Once version 8 (YOLOv8) and Single Shot Multibox Detector (SSD) architectures were developed. YOLOv8's simplified architecture enhances computational efficiency and inference speed during object identification. The SSD design includes the removal of conv5_x, avgpool, fc and softmax layers from the original model and the setting of all strides in conv4_x to 1 × 1. The backbone is followed by 5 additional convolutional layers, to which five detection heads are attached, and the sixth head is attached to the conv4_x layer. Experimental results show differences between dual and single networks, where the mean Average Precision ([email protected]) changes from 0.88 to 0.90. The unified model provides improvement in overall performance due to information fusion during object identification from vision and thermal imagery data streams. The most significant variation was observed when transitioning from YOLOv8 to SSD architecture, where YOLOv8 outperformed SSD by achieving higher [email protected] scores of 0.98 for the Harvester class and 0.94 for the Tractor class. Compared to SSD where [email protected] achieved 0.91 and 0.88, respectively.