Comparative evaluation of specialized deep learning and large multimodal models for image-based sea state recognition on the Beaufort scale

Recent advancements in large language models (LLMs) have demonstrated strong performance across a variety of language-based tasks. With the ability of LLMs to process multimodal inputs, including images, their potential applications are expanding. In this study, we explored this potential by focusing on a specialized task typically addressed by deep learning models. We fine-tuned the ResNet101d, XCIT-small, and Swin Base models to recognize sea states on the Beaufort scale (ranging from 1 to 8) using our dataset of images captured from ocean-going ships. Their performance was then compared to that of the OpenAI 4o model on a newly created test set. After manual calibration, the general LLM produced some encouraging results; however, the specialized deep neural network models demonstrated superior and more consistent performance across all metrics, along with faster real-time inference. The optimized transformer model Swin Base achieved the highest F1 score of 46.62%, while the calibrated GPT-4o model reached 27%. Likewise, the optimized Resnet101d achieved the lowest mean absolute error of 0.738, compared to 0.842 for the calibrated GPT-4o model.