The BSRA framework for dual sparse parameter efficient fine tuning with block structured gating and rank adaptation

Current large-scale language models face severe resource constraints when applied to downstream tasks. While full-fledged fine-tuning enhances model performance, it incurs substantial computational costs. Parameter-efficient fine-tuning (PEFT) techniques address resource limitations while maintaining high model efficiency; for instance, the emergence of low-rank adaptive methods like LoRA demonstrates outstanding parameter efficiency and model performance. However, fixed low-rank approaches struggle to adapt to varying parameter demands across layers, while unstructured sparsity, though capable of further parameter compression, often suffers from training instability and limited practical gains due to irregular shapes. To address this, this paper proposes BSRA, a dual-sparsity parameter-efficient fine-tuning framework integrating block-level structural sparsity with dynamic rank adaptation. BSRA achieves dual sparsity across rank and structural dimensions: at the coarse-grained level, it employs sensitivity-based dynamic rank pruning and non-linear cubic scheduling to smoothly remove low-contribution rank channels; at the fine-grained level, it introduces differentiable evolutionary block-level gating to structurally filter LoRA’s output features. To mitigate training oscillations during sparsification, we construct a unified constrained optimisation framework. This achieves synergistic optimisation of dual sparsity through distributed adjustment of rank scaling coefficients and continuous control of gated sparsity, enabling the model to adaptively converge towards a clear, stable low-rank structure. Experimental results demonstrate that BSRA matches or outperforms strong low-rank baselines with fewer trainable parameters, exhibiting outstanding parameter efficiency and task adaptability.