Improved Small Object Detection with YOLOv8 in Remote Sensing Imagery: Architecture Modifications and Loss Function Optimization

Remote sensing's ability to identify small objects is essential for earth observation, but algorithms developed for remote sensing remain challenged in achieving high accuracy. This research addresses critical challenges in remote sensing, including the small size of objects relative to the entire image, random object orientations requiring rotation-invariant detection, and significant variations in object scale. Furthermore, the model effectively handles noise occlusions and demonstrates the effectiveness of the Wise Intersection over Union (WIoU) loss function in improving performance and robustness. This research proposed a modified YOLOv8 model by replacing the C2f layer with the C3 layer, adding layers to the neck, and adding a head component. However, the modified model underperforms when using the Complete Intersection over Union (CIoU) loss function, achieving mean Average Precision (mAP) of 76.146% accuracy due to increased complexity, causing noisier predictions. In contrast, applying the WIoU loss function improves the model's performance, achieving an mAP of 77.183% accuracy and an improvement of 0.875% over the base model by better handling prediction variability.