FR ² ViT: Finetuning-free Token Reduction for Dense Prediction Through a Refinement-Reactivation Architecture

Token reduction is an efficient method for accelerating vision transformers. Techniques like token pruning and merging progressively decrease the number of active tokens to reduce the computation cost. However, when applied to dense prediction tasks, these techniques crudely cache low-level features or unfold non-self features to reconstruct the inactive tokens. This hard replacement results in significant information loss, especially at higher compression ratios. In this work, we propose FR<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>ViT, a finetuning-free refinement-reactivation architecture for efficient vision transformers, allowing tokens of varying informativeness to activate softly at differing costs. Specifically, we design a high-order informativeness assessment to partition tokens into superior and inferior categories accurately. We refine superior tokens through extensive attention interaction, while inferior tokens undergo effective reactivation, and we establish a lightweight interaction between two branches. Our approach ensures continuous and autonomous interaction of each token, and is finetuning-free to easily accommodate the large model era. Experiments across three dense prediction tasks demonstrate our method’s exceptional performance. For example, FR<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>ViT achieves a 1.95× FPS with only a 0.43% performance drop for Segmenter ViT-L.