Weakly supervised bird-flock counting in wetlands based on multimodal optical image perception

• Proposes a weakly supervised bird flock counting method in wetlands that integrates optical image features and visual semantic cues. • Develops a multimodal perception model, improving small object detection and adaptability to complex scenes and scale variations. • Introduces counting-specific visual prompts and a feature adapter to fuse visual semantics and optical features for enhanced counting accuracy. • Constructs the Wetland-Bird-Count dataset, with experiments showing the proposed method achieves near fully supervised performance. As crucial bio-indicators for wetland ecosystem health assessment, wetland birds play a pivotal role in ecological monitoring and conservation. This study address three challenges in avian population monitoring using optical remote sensing imagery, including high cost of manual annotation, difficulty in extracting small target features in complex background, and insufficient adaptability of multi-scale target recognition. We propose a weakly supervised bird-flock counting method based on the optical image multimodal perception model integrating optical image features and visual semantic features without location annotation. Based on optical image feature enhancement, visual semantic features related to the counting task are extracted through visual cues (counting text prompt), and a learnable feature adapter is introduced to fuse optical image features with visual semantic features. Thus, an optical image multimodal perception model with residual connection mechanism and multi-scale information interaction module is constructed. The residual connection mechanism effectively alleviates the interference caused by posture changes and complex background, and the multi-scale information interaction module solves the problem of target scale change through cross-scale semantic propagation. We construct an optical images bird-flock dataset named Wetland-Bird-Count for the Yellow River Delta coastal wetlands. The experimental results show that the MAE and MSE of the proposed method are 45.2 and 54.2, which is much more accurate than other weakly supervised and unsupervised methods and close to the fully supervised counting method, which verifies that the weakly supervised cluster counting using optical image visual cues can improve the accuracy of bird flock counting under lightweight annotation. This study provides a reliable quantitative analysis tool for optical image ecological monitoring.