A Uniform Multi-Modal Feature Extraction and Adaptive Local–Global Feature Fusion Structure for RGB-X Marine Animal Segmentation

Marine animal segmentation aims at segmenting marine animals in complex ocean scenes, which plays an important role in underwater intelligence research. Due to the complexity of underwater scenes, relying solely on a single RGB image or learning from a specific combination of multi-model information may not be very effective. Therefore, we propose a uniform multi-modal feature extraction and adaptive local–global feature fusion structure for RGB-X marine animal segmentation. It can be applicable to various situations such as RGB-D (RGB+depth) and RGB-O (RGB+optical flow) marine animal segmentation. Specifically, we first fine-tune the SAM encoder using parallel LoRA and adapters to separately extract RGB information and auxiliary information. Then, the Adaptive Local–Global Feature Fusion (ALGFF) module is proposed to progressively fuse multi-modal and multi-scale features in a simple and dynamical way. Experimental results on both RGB-D and RGB-O datasets demonstrate that our model achieves superior performance in underwater scene segmentation tasks.