Tracking with attention: A review of transformer-based object tracking

Traditional object tracking methods are often based on convolutional neural networks and handcrafted feature extraction techniques where they have seen remarkable success. However, these methods still face limitations in capturing global dependencies and contextual relationships in complex scenarios. Transformers, which were initially introduced to the field of natural language processing, have transfigured vision tasks by leveraging the self-attention mechanisms and global feature modeling capabilities. One of the tasks that has been most affected by the use of transformers is the object tracking task. This review explores the transformative impact of attention-based architectures in object tracking, and provides a comprehensive analysis of the current frameworks and their core principles. The ability of the attention mechanism to capture local and global dependencies and to associate queries between frames, has helped transformer-based models to achieve state-of-the-art performance. The utilization of transformers in object tracking has drastically increased over the past few years, initiating the new “tracking-by-attention” paradigm. This work focuses on different applications of transformer architecture in both single and multi-object tracking where each task is divided further by methodology. End-to-end approaches and hybrid fusion models that leverage additional data for tracking are also discussed. The models that are discussed, are categorized by their main approaches and transformer usage, and challenges such as computational cost and scalability are outlined, along with future research opportunities informed by successful methods. By examining recent advancements, this review is intended to advance understanding of transformer-based tracking capabilities and to promote continued innovation in this rapidly evolving field.