Towards a Ship Simulator Using Advanced Multi-Sensor Fusion for Maritime Navigation and Environmental Awareness

Abstract Accurate environmental sensing is essential for maritime applications, particularly in vessel tracking and obstacle avoidance within dynamic ocean environments. In this work, the existing SeaShips dataset is augmented to enhance ship detection and classification. A custom dataset, DetectShips, consisting of 14,000 real ship images, was developed and combined with the SeaShips dataset to train a computer vision algorithm integrated with a stereo sensor. The state-of-the-art detection model trained on this novel dataset was evaluated within a Unity-based simulation engine, demonstrating improved performance. The stereo cameras employed for ship detection and classification also facilitate precise depth estimation of target vessels relative to the host ship. Depth estimation accuracy is further enhanced by incorporating LiDAR data from the Unity platform. The data from both sensors are fused using a Kalman filter to achieve improved localization accuracy. The multi-vision sensor system effectively addresses the challenges of traditional optical sensing, including varying lighting conditions, fog, and occlusions. By integrating stereo camera outputs with LiDAR data, this hybrid approach ensures reliable performance in real-world maritime scenarios, delivering high-precision target localization even in complex oceanic environments.