D2VT: Better Detection and Description of Local Features with Vision Transformers

Constrained by the local nature of CNNs, existing local feature description methods often overlook global and contextual spatial information. Vision Transformers (ViT) address this by leveraging self-attention to capture long-range dependencies and preserve spatial details more effectively than CNNs. Our work introduces a hybrid architecture that merges CNNs for local feature extraction with ViT for global feature capture, enhancing performance across diverse vision tasks. We propose a novel hierarchical Transformer encoder adaptable to various image resolutions, yielding multi-scale features without positional encoding. Additionally, we introduce a consistent attention-weighted triple loss to get the attention map and to optimize and match local descriptors. Utilizing a feature pyramid, our method predicts keypoints at multiple scales, leading to improved localization accuracy. Experiments have shown that our approach is competitive with the leading contrastive learning methods in image matching benchmarks and demonstrates robust generalization in tasks like visual odometry.