Causal perception inspired representation learning for trustworthy image quality assessment

Despite great success in modeling visual perception, deep neural network based image quality assessment (IQA) still remains untrustworthy in real-world applications due to its vulnerability to adversarial perturbations. In this paper, we propose to build a trustworthy IQA model via Causal Perception inspired Representation Learning (CPRL). More specifically, we assume that each image is composed of Causal Perception Representation (CPR) and non-causal perception representation (N-CPR). CPR serves as the causation of the subjective quality label, which is invariant to the imperceptible adversarial perturbations. Inversely, N-CPR presents spurious associations with the subjective quality label, which may significantly change with the adversarial perturbations. We propose causal intervention to boost CPR and eliminate N-CPR. Specifically, we first generate a series of N-CPR intervention images, and then minimize the causal invariance loss. Then we propose a SortMask module to reduce Lipschitz and improve robustness. SortMask block small changes around the mean to eliminate N-CPR and can be plug-and-play. Experiments on four benchmark databases show that the proposed CPRL method outperforms many state-of-the-art methods and provides explicit model interpretation. To support reproducible scientific research, we release the code at https://clearlovewl.github.io . • We propose a Causal Perception Inspired Representation Learning (CPRL) for Trustworthy Image Quality Assessment. • Our method uses causal intervention to break false correlations (N-CPR) and find the true cause of image quality (CPR). • Specifically, to improve CPR, we first generate a series of N-CPR intervention images, and then minimize the causal invariance loss. To eliminate N-CPR, we propose a plug-and-play SortMask module to reduce Lipschitz constant by blocking small changes around the mean. • we conduct experiments on 8 baseline models and 5 attack methods. The results show the effectiveness of the CPRL strategy in improving the robustness of the baseline model against adversarial attacks. • To the best of our knowledge, this is the first time that a defense method for causal interpretation of NR-IQA models is proposed and theoretically supported.