A Multi-Layer Alignment and Adaptive Weighting Framework for Multi-Task Model Fusion

The integration of multi-task learning models presents challenges, particularly in addressing task-specific interference and enhancing cross-task knowledge transfer. To tackle these issues, we propose the Multi-Layer Task Alignment Weighted Fusion Model (MTAWM), a novel approach that optimizes model fusion by aligning tasks hierarchically and applying an adaptive weighting mechanism. MTAWM synchronizes task-specific layers using a Linear Mode Connectivity (LMC) strategy to prevent inter-layer parameter conflicts, while an adaptive weighting mechanism prioritizes task influence according to significance. Additionally, cross-layer consistency regularization is introduced to minimize deviations and stabilize the fusion process. Experimental evaluations demonstrate that MTAWM achieves superior performance in task accuracy, consistency, and model robustness when compared to existing baseline methods, positioning it as a robust solution for multi-task learning environments. MTAWM’s framework not only advances task alignment in multi-task settings but also holds potential for broader applications in multimodal data processing.