Learnable Counterfactual Attention for Music Classification

Counterfactual attention learning [1] utilizes counterfactual causality to guide attention learning and has demonstrated great potential in vision-based fine-grained recognition tasks. Despite its excellent performance, existing counterfactual attention is not learned directly from the network itself; instead, it relies on employing random attentions. To address the limitation and considering the inherent differences between visual and acoustic characteristics, we target music classification tasks and present a learnable counterfactual attention (LCA) mechanism, to enhance the ability of counterfactual attention to help identify fine-grained sounds. Specifically, our LCA mechanism is implemented by introducing a counterfactual attention branch into the original attention-based deep-net model. Guided by multiple well-designed loss functions, the model pushes the counterfactual attention branch to uncover biased attention regions that are meaningful yet not overly discriminative (seemingly accurate but ultimately misleading), while guiding the main branch to deviate from those regions, thereby focusing attention on discriminative regions to learn task-specific features in fine-grained sounds. Evaluations on the benchmark datasets <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">artist20</i> [2], <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">GTZAN</i> [3], and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FMA</i> [4] demonstrate that our LCA mechanism brings a comprehensive performance improvement for deep-net models on singer identification and musical genre classification. Moreover, since the LCA mechanism is only used during training, it doesn't impact testing efficiency.