Multimodal object tracking using raw visible and thermal infrared data

Visual Object Tracking (VOT) in computer vision involves estimating an object’s position in a sequence of images, starting from its initial position. However, environmental factors such as poor lighting can hinder VOT algorithms. A solution is multi-modal approaches, particularly combining visible and thermal imaging (RGB-T). Vision Transformers have significantly improved RGB-T tracking algorithms. However, their increased number of parameters has led to longer inference times and reduced embedded capabilities. This study evaluates the feasibility of using an alternative color space for the visible modality as a way to reduce the size of RGB-T tracking algorithms and assess the impact of such changes on tracking performance. We propose a Bayer-Thermal network architecture based on Vision Transformers and demonstrate that when trained with a specific cross-color space distillation technique, yields promising results with our top-performing model having a 67% model size reduction compared to the state of the art. While our preliminary results are encouraging, we do not reach state of the art performances in terms of robustness. Thus, future work should focus on developing a specialized dataset for training and evaluation of Bayer-Thermal tracking models.