Toward a Ship Simulator Using Advanced Multisensor Fusion for Maritime Navigation and Environmental Awareness

Abstract Accurate environmental sensing is essential for maritime applications, particularly in vessel tracking and obstacle avoidance within dynamic ocean environments. Existing maritime vision datasets (e.g., SeaShips, HRSC2016) lack sufficient variation in viewpoints and weather scenarios, limiting the real-world robustness and generalization of current deep learning models. In this work, the existing SeaShips dataset is augmented to enhance ship detection and classification. In particular, a novel custom dataset, i.e., DetectShips, consisting of 14,000 real ship images, is developed and combined with the SeaShips dataset to train a computer vision algorithm integrated with a stereo sensor. The state-of-the-art detection model trained on this novel dataset is evaluated within a Unity-based simulation engine, demonstrating improved performance. A hybrid sensing framework, StereoLiD, that integrates stereo vision systems with LiDAR is also presented in this article. The stereo cameras employed for ship detection and classification facilitate precise depth estimation of target vessels relative to the host ship. Depth estimation accuracy is further enhanced by incorporating LiDAR data from the Unity platform. The data from both sensors are fused using a Kalman filter to achieve improved localization accuracy. The multivision sensor system effectively addresses the challenges of traditional optical sensing, including varying lighting conditions and fog. By integrating stereo camera outputs with LiDAR data, this hybrid approach ensures reliable performance in real-world maritime scenarios, delivering high-precision target localization even in complex oceanic environments.