Maritime surveillance using unmanned vehicles: deep learning-based vessel re-identification

Maritime surveillance is crucial for ensuring compliance with regulations and protecting critical maritime infrastructure. Conventional tracking systems, such as AIS or LRIT, are susceptible to manipulation as they can be switched off or altered. To address this vulnerability, there is a growing need for a visual monitoring system facilitated by unmanned systems such as unmanned aerial vehicles (UAVs) and unmanned surface vehicles (USVs). Equipped with sensors and cameras, these unmanned vehicles collect vast amounts of data that often demand time-consuming manual processing. This study presents a robust method for automatic target vessel re-identification from RGB imagery captured by unmanned vehicles. Our approach uniquely combines visual appearance and textual data recognized from the acquired images to enhance the accuracy of target vessel identification and authentication against a known vessel database. We achieve this through utilizing Convolutional Neural Network (CNN) embeddings and Optical Character Recognition (OCR) data, extracted from the vessel’s images. This multi-modal approach surpasses the limitations of methods relying solely on visual or textual information. The proposed prototype was evaluated on two distinct datasets. The first dataset contains small vessels without textual data and serves to test the performance of the fine-tuned CNN model in identifying target vessels, trained with a triplet loss function. The second dataset encompasses medium and large-sized vessels amidst challenging conditions, highlighting the advantage of fusing OCR data with CNN embeddings. The results demonstrate the feasibility of a computer vision model that combines OCR data with CNN embeddings for target vessel identification, resulting in significantly enhanced robustness and classification accuracy. The proposed methodology holds promise for advancing the capabilities of autonomous visual monitoring systems deployed by unmanned vehicles, offering a resilient and effective solution for maritime surveillance.