NOT-156: Night Object Tracking Using Low-Light and Thermal Infrared: From Multimodal Common-Aperture Camera to Benchmark Datasets

Night object tracking (NOT) is aimed at tracking objects under low-illumination conditions at night. Existing works concentrate on thermal infrared modality, while some RGB and thermal infrared (RGB-T) data also contain night scenes. However, night scenes in these datasets are mostly well-lit, making it challenging to fully cover low-illumination scenarios. In this article, we focus on the NOT task and build up a novel low-light visible and thermal infrared (LOL-T) multimodal benchmark dataset for NOT-156. To achieve night vision, we design a common-aperture LOL-T camera by integrating a highly dynamic low-light visible imaging sensor with a thermal infrared sensor in a common aperture optical system. The proposed dataset consists of 156 video sequences and a total of 170k annotated frames, including various low-illumination night scenes such as dark rooms, streets, corridors, and so on. Compared with existing datasets, NOT-156 has more comprehensive and distinctive attributes (thermal variation, noise, high illumination overexposure, etc.). Comprehensive experiments are carried out to evaluate the performance of the advanced visible, infrared, and visible-thermal trackers on the proposed NOT-156 dataset. The authors believe that NOT-156 has great potential in the application and development of night vision. The dataset will be made available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://rsidea.whu.edu.cn/NOT156_dataset.htm</uri>.