A novel BH3DNet method for identifying pine wilt disease in Masson pine fusing UAS hyperspectral imagery and LiDAR data

• Combining hyperspectral and LiDAR data to detect pine wilt disease. • Proposing an instance segmentation network, BH3DNet, based on an improved PointNet++. • Accurately identifying individual trees at different stages of PWD infection. Pine Wilt Disease (PWD) is a forest infectious disease that inflicts substantial economic losses to China’s forestry. Its rapid spread and the significant challenges associated with its control make early detection of infected trees crucial for disaster prevention. Unmanned aerial systems (UASs) hyperspectral imaging (HSI) and light detection and ranging (LiDAR) technologies provide high-resolution spectral diagnostic information coupled with intricate three-dimensional structural data, which has potential for fine grained monitoring of PWD. However, how to fuse HSI and LiDAR data to identify the early infected individual trees is still a challenge. This study presents a novel instance segmentation network, BH3DNet, to identify individual trees at different PWD-infected stages by extracting high-level abstract features based on the fusion of drone HSI and LiDAR data. BH3DNet introduces the PointNet++ model as the base network, and incorporates a shared encoder and twin parallel decoders to align semantic category prediction and instance segmentation of individual trees in an end-to-end approach. By applying an enhanced point cloud dataset that fuses drone HSI and LiDAR point cloud data, this model facilitates the identification of PWD infection stages at the individual tree scale. We evaluated the proposed model in a Masson pine forest stand sparsely mixed with broadleaf trees in a variety of infection states ranging from healthy to severely infected by PWD, and compared the performance of the model using the RGB bands, full HSI bands and screened bands as inputs, respectively. BH3DNet achieves an overall accuracy of 89.65 % with a Kappa × 100 of 87.29 for identifying individual trees using screened HSI bands and LiDAR point cloud, significantly outperforming the Mask R-CNN using only HSI data (overall accuracy: 70.81 %, Kappa × 100: 64.16). Moreover, BH3DNet’s accuracy at the early infection stage reaches 83.75 %. It proves that fusing HSI and point cloud data reflects the information of individual trees distribution and infection status, and the BH3DNet is suitable for high-precision monitoring of PWD.