Prompt-Conditioned Channel Attention for Hierarchical Feature Modulation toward Anatomy-Agnostic Segmentation

Abstract Accurate and anatomically plausible medical image segmentation remains challenging due to low contrast, ambiguous boundaries, interpatient variability, and modality-specific artifacts. Interactive segmentation using spatial prompts has emerged as a promising strategyto guide attention and improve localization, particularly in low-contrast or structurally ambiguous regions. However, existing methodstypically restrict prompt integration to late-stage fusion and lack explicit mechanisms for channel-wise feature modulation, limiting theirability to capture deeper contextual and modality-specific variations across hierarchical representations. To address these limitations, weintroduce Prompt-Conditioned Channel Attention (PCCA), a novel modulation mechanism that enables deep, hierarchical integration ofsemantic prompts within encoder-decoder segmentation networks. PCCA extracts compact channel descriptors from both image and promptfeatures via pooling, projects them into a shared latent space, and fuses them through a gated excitation unit to compute prompt-awarechannel attention weights. These weights perform hierarchical channel-wise attention by adaptively reweighting feature responses atevery network stage, enabling spatially selective and semantically enriched representations. Building on this mechanism, we proposePROMISE-Net (PROmpt Modulated Integration for SEgmentation), instantiated in two architectural variants: a convolutional model(PROMISE-CNN) and a transformer-based model (PROMISE-Txformer). Across ISIC-Lesion, Kvasir-Polyp, CAMUS-Cardiac, andKvasir-Instrument benchmarks, integrating PCCA into PROMISE-CNN yielded relative IoU improvements of 10.4%, 8.7%, 0.8%, and3.4% over baseline U-Net, while PROMISE-Txformer achieved corresponding gains of 7.6%, 23%, 2.1%, and 1.1% over baseline UNETR.These results demonstrate consistent cross-architectural, cross-modal, and cross-anatomical improvements, establishing PCCA coupledPROMISE-Net as a scalable and generalizable framework for prompt-aware feature modulation in medical image segmentation. The codeis available at https://github.com/kamruleee51/PROMISENet.