Compactness-aware proposal refinement for foundation model-assisted maize instance annotation in complex field environments

Introduction Instance segmentation in complex field scenes depends on accurate pixel-level labels, but tracing every plant is slow and limits the scale of agricultural vision datasets. We examined a less studied source of error in automated annotation: the spatial uncertainty of open-vocabulary detection boxes used as prompts for a foundation segmentation model. Methods The resulting workflow combines Grounding DINO, Compactness-Aware Proposal Refinement (CAPR), SAM2, and manual refinement. Experiments used 695 field images with 6,269 manually annotated maize instances. Results Among 4,680 valid proposal-GT-mask triplets, Proposal Box IoU was positively associated with SAM2 Mask IoU (Spearman rho = 0.380, p< 0.001). Within 239 non-duplicated GT-level pairs, the better-localized proposal also produced the better mask in 234 cases (97.9%), with a mean Mask IoU difference of 8.86 percentage points. A controlled perturbation analysis using one maize instance from each of the 695 images further showed that box expansion and contraction produced opposite Precision–Recall responses, confirming the sensitivity of SAM2 output to box-prompt geometry. CAPR re-ranks locally overlapping proposals by combining detection confidence with relative compactness. Of 69 valid cases in which CAPR and Standard NMS selected different proposals for the same GT instance, CAPR chose the box with higher IoU in 65 (94.2%) and the linked SAM2 mask with higher IoU in 63 (91.3%). YOLO26n-seg trained on CAPR + SAM2 labels reached 81.10% Mask mAP50 and 43.52% Mask mAP50–95. Manual refinement increased these values to 91.94% and 52.93%, close to 93.16% and 53.84% with Manual GT. In a single-annotator comparison, human operation time fell from 275 to 85 s per image, a reduction of 69.09% and a 3.24-fold increase in annotation throughput per unit of human time. Discussion Proposal geometry influences the quality of SAM2 prompts for seedling maize. Combining selective proposal refinement with human review reduced annotation effort and retained much of the downstream training value of Manual GT.