Behavior Recognition of Ship Pilots Based on Multimodal Video Data

The irregular behavior of ship pilots poses a safety hazard in navigation. Identifying abnormal behaviors, reducing water traffic accidents caused by human factors, and improving navigation safety supervision have become critical issues. To better simulate real-world scenarios, this study has constructed a private dataset using a full-field-of-view simulated bridge, covering six actions with a total of 1,560 videos. This dataset introduces several innovations in the field of behavior recognition. The study combines RGB, infrared (IR), and depth (Depth) multimodal data, addressing the limitations of single modalities being easily affected by lighting and lacking three-dimensional information, thus improving the robustness and accuracy of recognition. We have designed a lightweight 3D CNN feature extraction network, which ensures efficient feature extraction while reducing computational cost, making it suitable for resource-limited scenarios. Additionally, this study introduces a dual attention mechanism for multimodal fusion, enhancing feature representation by combining features from different modalities and leveraging their complementarity and correlation. This approach has enabled the model to achieve an accuracy of 92.95% on the custom dataset, with recognition rates for texting and using a telescope reaching 98% and 94%, respectively, and an F1-score of 0.8731.