SAF-Net: Stage-Aware Semantic Allocation with Frequency-Refined Vision-Language Decoding for Medical Lesion Segmentation

Medical lesion segmentation remains challenging due to ambiguous boundaries, heterogeneous appearances, and the limited ability of visual features alone to capture clinically meaningful semantics. Existing vision-language segmentation methods have demonstrated the value of textual guidance, but they often rely on fixed or shared textual representations across different decoder stages, despite the fact that visual features progressively evolve from coarse semantic representations to finer spatial details. To address this stage-wise representation mismatch, we propose SAF-Net, a stage-aware vision-language segmentation framework that couples stage-specific semantic allocation with frequency-refined progressive decoding. Specifically, Adaptive Hierarchical Text Allocation (AHTA) retains the hidden representations from all layers of a frozen medical language encoder and learns independent layer-mixing distributions for different text-guided decoder stages, enabling stage-specific semantic guidance without manually predefined layer-to-stage correspondence. Meanwhile, Frequency-Gated Spatial-aware Channel Attention (FGSCA) refines the incoming decoder features through complementary frequency-aware spatial-channel modeling before cross-modal interaction. By coordinating hierarchical textual semantics with frequency-aware visual refinement across progressive decoding stages, SAF-Net allows the representations participating in multimodal interaction to better adapt to the changing requirements of hierarchical decoding. Extensive experiments on QaTa-COV19 and MosMedData+ demonstrate the effectiveness and competitiveness of the proposed framework.