Debiasify: Self-Distillation for Unsupervised Bias Mitigation

Simplicity bias is a critical challenge in neural net-works since it often leads to favoring simpler solutions and learning unintended decision rules captured by spuri-ous correlations, causing models to be biased and dimin-ishing their generalizability. While existing solutions rely on human supervision, obtaining annotations of the dif-ferent bias attributes is often impractical. To tackle this, we present Debiasify, a novel self-distillation approach that works without any prior information about the nature of biases. Our method leverages a new distillation loss to distill knowledge within a network; from a deep layer where complex, highly-predictive features reside, to a shallow layer where simpler yet attribute-conditioned features are found in an unsupervised manner. In this way, Debiasify learns robust, debiased representations that generalize well across various biases and datasets, enhancing worst-group performance and overall accuracy. Extensive experiments on computer vision and medical imaging benchmarks show the efficacy of our method, significantly outperforming the previous unsupervised debiasing methods (e.g., a 10.13% improvement in worst-group accuracy on Wavy Hair classi-fication in CelebA) while achieving comparable or superior performance to supervised methods. Our code is publicly available at the following link:Debiasify.