LBTR: local feature mining and background activation minimization in transformers for weakly supervised object localization

Weakly supervised object localization (WSOL) aims to learn object localizers using only image-level labels. Traditional WSOL methods based on convolutional neural networks (CNNs) tend to focus on the most discriminative regions of the object, often overlooking its full extent. In contrast, Visual Transformers, by exploiting long-range feature dependencies within the self-attention mechanism, enables more comprehensive localization of the entire object. However, transformers lack the local feature extraction capabilities inherent to CNNs, which can compromise the preservation of fine local details in WSOL. In addition, current transformer-based WSOL methods rely on cross-entropy to guide the classifier in generating localization maps, leading to conflicts between the classification and localization tasks. We proposed local feature mining and background activation minimization in transformers for WSOL. Specifically, we first propose an efficient cross-space neighborhood interaction attention module, designed to effectively capture local features while maintaining the ability to integrate global information. In addition, we introduce a mechanism for generating foreground prediction maps and optimize the generator’s learning through a background activation minimization strategy to refine these maps and capture more extensive object regions. Finally, extensive experiments on two widely used datasets, CUB-200-2011 and ILSVRC, validate the effectiveness of our method. The results consistently demonstrate significant improvements in object localization performance.