Maritime object tracking from a commercial vessel using radars, electro-optical cameras and navigational instruments

• Tracking using JIPDA with an additional Poisson/PHD filter for the undetected objects. • Application to data from a large commercial vessel in operation along coast of Norway. • Heterogeneous sensors; radars, electro-optical cameras and navigational instruments. • Introduction and demonstration of a visibility state per sensor type. The aim of this work was to develop partial maritime situational awareness based on a tracking system using two radars, 12 cameras and common navigational instruments mounted on a commercial vessel operating on the coast of Norway. Objects on the sea surface are tracked based on detections extracted from sensor images which are segmented by convolutional neural networks. The tracker structure consists of a JIPDA filter where marginalized association probabilities are approximated using loopy belief propagation and track initialization is performed using intensities from a PHD filter. The novelties of the proposed tracker are: a) the application to data from a large commercial vessel in operation, b) the use of heterogeneous sensors (multiple radars and optical cameras), and c) the introduction of a visibility state per sensor type enabling the tracker to maintain tracks for objects that are invisible to one of the sensor types. Although no ground truth was available, the results indicate that the implementation efficiently confirms true tracks, discards false tracks, finds and maintains tracks on objects that are (temporarily) invisible to one of the sensor types, and provides increased accuracy compared to a single sensor setup. Video of tracking results available at: https://vimeo.com/534840034 .