Recursive Counterfactual Deconfounding for image recognition

Image recognition is a classic and common task that has been widely applied in computer vision in the past decade. Most existing methods in literature aim to learn discriminative features for classification from labeled images; however, they generally neglect confounders that infiltrate the learned features, resulting in low discrimination performances on test images. To address this problem, we propose a Recursive Counterfactual Deconfounding model for image recognition in both closed- and open-set scenarios based on counterfactual analysis, called RCD. The proposed model consists of a factual graph and a counterfactual graph, where the relationships among image features, model predictions, and confounders are built and updated recursively to learn more discriminative features. It performs recursively so that subtler counterfactual features can be learned and progressively eliminated. Both the discriminability and generalizability of the proposed model can be improved accordingly. In addition, a negative correlation constraint is designed to alleviate the negative effects of counterfactual features during model training. Extensive experimental results in both the closed-set and open-set image recognition tasks demonstrate the effectiveness of the proposed method. Particularly under the hard testing mode of the three public datasets (Aircraft, CUB and ImageNet), the proposed method obtains a higher AUROC (Area Under the Receiver Operating Characteristic curves) by 0.7%–3.5% and a higher OSCR (Open-Set Classification Rate) by 0.2%–4.0%. • We propose a deconfounding model for alleviating the confounders that infiltrate into the features learned by a typical image recognition network based on causal analysis called Recursive Counterfactual Deconfounding model (RCD), where a learnable strategy is designed for dynamically obtaining the counterfactual features, and a recursive manner is used for progressively learning subtler counterfactual features. • We design a negative correlation constraint between the discriminative features and the counterfactual features at the model training stage, which further alleviates the negative effects of the counterfactual features. • We conduct extensive experiments in both closed-set and open-set scenarios for evaluating the effectiveness of the proposed RCD model. Results in Section 4 demonstrate that the proposed method can significantly improve different backbones, and it also outperforms the 11 state-of-the-art baselines.