Frozen vision transformers can be reused via cross-scale feature fusion

Abstract As Vision Transformer (ViT) models continue to scale, the training and fine-tuning costs for achieving advanced performance have surged, forcing us to rethink the sustainability of relying solely on model scaling. In this work, we explore a more cost-effective alternative. By effectively exploiting the complementary potential of existing frozen models, we avoid the expense of training large models from scratch. Based on this idea, we propose the Dual-ViT Fusion Transformer (DVFT). This framework explicitly aligns and fuses heterogeneous features from different frozen backbones, thereby addressing the limited representational scope and insufficient cross-scale semantic coverage of a single frozen model. Such limitations are difficult to mitigate within existing parameter-efficient fine-tuning frameworks, which are primarily designed for single-model settings. Specifically, DVFT introduces two complementary guidance paradigms. The first is Intrinsic Anchor-based calibration, which uses the frozen models’ own global semantic representations as alignment references to establish cross-model semantic consistency. The second is Extrinsic Query-based guidance, which employs learnable queries as an information bottleneck to progressively extract and integrate complementary discriminative features across models, enabling flexible feature interaction. Experimental results show that our method approaches the performance of expensively trained models with very few training epochs. These findings indicate that DVFT provides a scalable and low-cost pathway for efficiently reusing existing frozen models to achieve high-performance visual recognition.