Forest Vegetation 3D Localization Using Deep Learning Object Detectors

Forest fires are becoming increasingly prevalent and destructive in many regions of the world, posing significant threats to biodiversity, ecosystems, human settlements, climate, and the economy. The United States of America (USA), Australia, Canada, Greece and Portugal are five regions that have experienced enormous forest fires. One way to reduce the size and rage of forest fires is by decreasing the amount of flammable material in forests. This can be achieved using autonomous Unmanned Ground Vehicles (UGVs) specialized in vegetation cutting and equipped with Artificial Intelligence (AI) algorithms to identify and differentiate between vegetation that should be preserved and material that should be removed as potential fire fuel. In this paper, an innovative study of forest vegetation detection, classification and 3D localization using ground vehicles’ RGB and depth images is presented to support autonomous forest cleaning operations to prevent fires. The presented work, which is a continuation of a previous research, presents a method for 3D objects localization in the real-world using Deep Learning Object Detection (DLOD) combined with an RGB-D camera. It presents and compares results of eight recent high-performance DLOD architectures, YOLOv5, YOLOv7, YOLOv8, YOLO-NAS, YOLOv9, YOLOv10, YOLO11 and YOLOv12, to detect and classify forest vegetation in five classes: “Grass”, “Live vegetation”, “Cut vegetation”, “Dead vegetation”, and “Tree-trunk”. For the training of the DLOD models, our custom dataset acquired in dense forests in Portugal is used. A methodology that combines the best DLOD trained for vegetation detection and classification and an RGB-D camera for the 3D localization of the classified detected objects in the real-world. The presented methods are employed in an Unmanned Ground Vehicle (UGV) to localize forest vegetation that needs to be thinned for fire prevention purposes. A key challenge for autonomous forest vegetation cleaning is the reliable discrimination of objects that need to be identified to reach the goal of fire prevention using autonomous unmanned ground vehicles in dense forests. With the obtained results, forest vegetation is precisely detected, classified and localized using the DL models and the localization method presented. Also, the fastest DLOD architecture to train is YOLOv5, and the fastest to infer are YOLOv7 and YOLOv12. The innovation presented is the detection, classification, and 3D localization of the vegetation using DLOD architectures, in real-time, with a localization error of the real-world object in width, height and depth under 21.4, 20.7 and 11%, respectively, using only a depth camera and a processing unit. The 3D localized objects are defined as parallelepiped geometrical shapes. The methodology for vegetation detection, classification and localization presented in this paper is highly suitable for future autonomous forest vegetation cleansing, specialized using unmanned ground vehicles.