Cross-domain analysis of YOLOv8 and faster R-CNN models for enhanced precision in maritime object detection

Recent advancements in machine vision, particularly through Convolutional Neural Networks (CNNs), have significantly improved object detection in maritime environments. This study evaluates the performance of two leading object detection algorithms—YOLOv8 and Faster R-CNN—along with their respective variants, across two specialized maritime datasets: SeaShip and the Sea Maritime Dataset (SMD). Using comprehensive intra-domain and cross-domain evaluations, we analyzed precision, recall, and mean Average Precision (mAP) metrics over 50 training epochs. The YOLOv8x variant demonstrated exceptional adaptability to the SMD dataset, achieving high precision and recall rates of 98.3% and 96.1%, respectively. Meanwhile, the YOLOv8m variant performed more effectively on the SeaShip dataset. Faster R-CNN with the X101-FPN backbone showed comparable results to YOLO in intra-domain evaluations but outperformed YOLO in cross-domain scenarios. Specifically, for the SMD dataset, Faster R-CNN improved the mAP(50) score by 47.9%, while for the SeaShip dataset, it achieved a 4.48% improvement. This paper highlights the challenges of deploying machine vision in maritime contexts, where environmental variability and dataset specificity play critical roles. Cross-domain analysis revealed substantial performance degradation when models were applied outside their training domain, underscoring the importance of robust domain adaptation strategies. Overall, the findings emphasize the need to carefully select object detection algorithms tailored to dataset characteristics in order to optimize performance across diverse maritime environments.