Todo Fir Crown Instance Segmentation in Dense Plantation Forest Using Polar-FFT and Treetop Queries

Abstract. Monitoring individual trees from unmanned aerial vehicle (UAV)-derived orthomosaics and digital surface models (DSMs) is important for forest management, but instance segmentation remains difficult in dense planted forests in Japan, where Structure from Motion (SfM)-derived DSMs often exhibit blurred crown boundaries, noise, and substantial variation in quality. Existing approaches can benefit from treetop information, but they are sensitive to threshold selection and may introduce false positives when incorrect treetop candidates are provided. To address this problem, we propose a method based on the polar coordinate transform and the fast Fourier transform (PFFT) that represents the local DSM shape around treetop candidates as compact descriptors and integrates them into Mask2Former. The descriptors are used both to suppress low-reliability candidates and to provide spatially meaningful treetop queries to the decoder, thereby improving the separation of adjacent crowns. We evaluated the method on Abies sachalinensis (Todo fir) plantation data acquired at two sites in Hokkaido, Japan, using Mask R-CNN and Mask2Former as baselines. Compared with standard Mask2Former, the proposed method improved mAP50 from 90.14% to 90.87%, mAP75 from 52.18% to 55.47%, and the F1 score at a confidence threshold of 0.5 from 89.86% to 92.08%. It also reduced the number of false positives by 41% without increasing false negatives. Qualitative results showed fewer over-merged crowns and better crown-boundary delineation. These results indicate that treetop-centered local shape cues are effective for instance segmentation in densely planted forests, although further validation across additional species and regions is needed.