Hybrid vision system for enhanced situational awareness in Unmanned Surface Vehicles: Decision-Level Camera-LiDAR fusion with supervised and unsupervised approaches

The integration of LiDAR and cameras with an efficient data fusion approach significantly improves Enhanced Situational Awareness (ESA) for small-sized Unmanned Surface Vehicles (USVs). This is critical for early obstacle detection, particularly when maritime obstacles are only 5 to 10 seconds away. This is important to ensure effective collision avoidance in regions without reliable Global Positioning System (GPS) and Automatic Identification System (AIS), while maintaining low computational requirements and real-time performance. This paper presents a system aimed at ESA using an unsupervised approach and a trained deep learning model to detect multiple maritime obstacles in denied GPS zones. The proposed system incorporates a technical mechanism that uses a compact hardware system with an integrated advanced computing module and sensors for small-sized USVs. It addresses the challenges of achieving peak performance on an edge machine learning computer integration to reduce computational overhead. Further, it minimizes temporal detection variations using sophisticated filters and clustering in dynamic maritime environments with synchronized LiDAR and camera data fusion. The detection model, trained using Maritime Federated Large Dataset (MFLD2), achieved over 99% operational accuracy with the proposed data fusion approach. The system’s capacity to precisely identify obstacle location and distance is validated by experimental findings, enabling real-time situational awareness.