Enhancing Instance Segmentation in High-Resolution Images Using Slicing-Aided Hyper Inference and Spatial Mask Merging Optimized via R-Tree Indexing

Instance segmentation in high-resolution images is essential for applications such as remote sensing, medical imaging, and precision agriculture, yet remains challenging due to factors such as small object sizes, irregular shapes, and occlusions. Tiling-based approaches, such as Slicing-Aided Hyper Inference (SAHI), alleviate some of these challenges by processing smaller patches but introduce border artifacts and increased computational cost. Overlapping tiles can mitigate certain boundary effects but often result in duplicate detections and boundary inconsistencies, particularly along patch edges. Conventional deduplication techniques, including Non-Maximum Suppression (NMS) and Non-Mask Merging (NMM), rely on Intersection over Union (IoU) thresholds and frequently fail to merge fragmented or adjacent masks with low mutual IoU that nonetheless correspond to the same object. To address deduplication and mask fragmentation, Spatial Mask Merging (SMM) is proposed as a graph clustering approach that integrates pixel-level overlap and boundary distance metrics while using R-tree indexing for efficient candidate retrieval. SMM was evaluated on the iSAID benchmark using standard segmentation metrics, with tile overlap configurations systematically examined to determine the optimal setting for segmentation accuracy. The method achieved a nearly 7% increase in precision, with consistent gains in F1 score and Panoptic Quality over existing approaches. The integration of R-tree indexing facilitated faster candidate retrieval, enabling computational performance improvements over standard merging algorithms alongside the observed accuracy gains.