A port navigation perception method via shipborne camera-guided light detection and ranging fusion

The autonomous navigation of Maritime Autonomous Surface Ships (MASS) highly depend on perception performance. This study utilizes sensors such as shipborne Light Detection and Ranging (LiDAR), Camera, Inertial Measurement Unit (IMU), and Real-Time Kinematic (RTK) to propose a ship perception system based on multimodal data fusion named Shipborne LiDAR-Camera Fusion (SLC-Fusion). In the frontend, a multi-sensor collaborative calibration method based on the iKalibr framework is improved, and adaptive point cloud motion compensation is achieved by combining IMU data. In the backend, the shipborne LiDAR denoising approach based on intensity features is proposed to improve the quality of raw data. Subsequently, the point cloud features after dimensionality reduction are fused with optical images, and an Artificial Intelligence (AI) detection module based on You Only Look Once (YOLO) 11s-seg is used to acquire target semantic and effective pixel indices. Simultaneously, a lightweight port target clustering approach based on local spatial features is proposed to obtain the three-dimensional (3D) states. Finally, the latitude and longitude information of targets are calculated by fusing RTK data. On this basis, this study establishes a perception dataset for port navigation. Real-world experiments demonstrate that SLC-Fusion possesses excellent ship perception performance. The minimum ranging error is 0.07 m (m), with a mean Intersection over Union (mIoU) of 67.0%, an average Mean Absolute Error (MAE) of 0.34 m, and a Root Mean Square Error (RMSE) of 0.47 m. The average processing time for a single frame of images and point clouds is 5.9 ms (ms) and 4.6 ms, respectively.