YOLO-GIR: A new insect detection algorithm using inception-based variable-size convolutions

• New variable Inception-based bottleneck structure and Inception-based lightweight feature encoded model was proposed named as YOLO-GIR. • Newly designed a low-cost and easy-to-install light-induced based insect detection system was developed. • Tiny Insect image dataset from greenhouses was collected and potential to serve as open-source datasets. • YOLO-GIR model achieved a 6.7% improvement in mAP50 and an 8.1% improvement in mAP50–95 over the latest baseline YOLOv11s. • YOLO-GIRs reduced the number of GFLOPs by 55.2% compared with YOLOv8m and enhanced robustness to background noise, reduced false positives. Tiny insects and reproduction during hydroponic cultivation in greenhouses are critical early indicators of environmental imbalance. This study presents an automated insect detection system based on a deep learning system and quantifies insect clustering in real time. To enhance detection performance, particularly for small insect targets, we propose YOLO-GIR and lightweight version YOLO-GIRe for edge device, an improved version of the YOLOv8s model developed as a new contribution to the YOLO series for real-time applications. The proposed model introduces a novel variable Inception-based convolutional bottleneck (VI-bottleneck) and a newly designed C2F-VC structure, replacing the original C2f module to strengthen multiscale feature learning. Additionally, an Inception-based lightweight feature encoder (I-LFE) was integrated before the neck module to improve the model’s capacity for lightweight and efficient representation. The experimental results on a custom insect dataset demonstrated that YOLO-GIR achieved an mAP50 of 92.2%, which was a 3.9% improvement over YOLOv8s, and an mAP50–90 of 68.8%, which was a 7.8% improvement over YOLOv8s. Moreover, YOLO-GIR achieves a 6.7% improvement in mAP50 and an 13.2% improvement in mAP50–95 over the latest baseline YOLOv11s. On the same detection platform, YOLO-GIR consistently outperforms existing algorithms in both accuracy and stability. During the automated monitoring period, the system provided reliable indicators of greenhouse environmental trends, demonstrating its practical applicability as a low-cost and effective tool for integrated pest management in controlled and outdoor environments for small insect detection.