3DM: Distill, Dynamic Drop, and Merge for Debiasing Multi-modal Large Language Models

The rapid advancement of Multi-modal Language Models (MLLMs) has significantly enhanced performance in multimodal tasks, yet these models often exhibit inherent biases that compromise their reliability and fairness.Traditional debiasing methods face a trade-off between the need for extensive labeled datasets and high computational costs.Model merging, which efficiently combines multiple models into a single one, offers a promising alternative but its usage is limited to MLLMs with the same architecture.We propose 3DM, a novel framework integrating Distill, Dynamic Drop, and Merge to address these challenges.3DM employs knowledge distillation to harmonize models with divergent architectures and introduces a dynamic dropping strategy that assigns parameter-specific drop rates based on their contributions to bias and overall performance.This approach preserves critical weights while mitigating biases, as validated on the MMSD2.0sarcasm detection dataset.Our key contributions include architecture-agnostic merging, dynamic dropping, and the introduction of the Bias Ratio (BR) metric for systematic bias assessment.Empirical results demonstrate that 3DM outperforms existing methods in balancing debiasing and enhancing the overall performance, offering a practical and scalable solution for deploying fair and efficient MLLMs in real-world applications.The code of this paper can be found at https://github.com/JesseZZZZZ/3DM.