COFP: A Collaborative Optimization Framework With Polyhedral Feature Extraction for Multi-Weather Image Restoration

Image restoration in adverse weather conditions is a critical research focus in computer vision and autonomous driving. In this work, we introduce COFP, a collaborative optimization framework designed to simultaneously enhance the performance of image de-raining, de-snowing, and de-hazing tasks across diverse datasets. The core of COFP lies in its adaptive optimization of weather-shared and weather-specific parameters, enabling the extraction of polyhedral features that effectively integrate both weather-shared and weather-specific attributes, thus substantially boosting multi-weather performance of the network. Technically, we design a polyhedral feature extraction module (PFEM) to facilitate the acquisition of weather-shared attribute and weather-specific attribute. In PFEM, we first introduce an element-adaptive sharing strategy (ESS) that dynamically activates either weather-shared or weather-specific parameters for each element of the weight matrix based on learnable scores, thereby adaptively determining which parameters are shared or not. Secondly, we develop a feature extraction enhancement strategy (FEES), which extends two pathways in PFEM comprised of standard convolutional layers to enhance the capture of polyhedral features, further promoting the model performance. Furthermore, we propose a gradient balancing algorithm (GBA) that mitigates the unequal competition among tasks for shared parameters during network optimization by adaptively adjusting the direction of task gradients, effectively addressing the negative transfer issue induced by domain variations in multi-task learning. Experimental results demonstrate that COFP delivers state-of-the-art performances across various adverse weather image restoration benchmarks.