Enhancing Port Security: A Dual-Stage Multimodal Agent for Reducing False Alarms in Human–Vehicle Interaction Detection

In the field of port security, traditional human–vehicle interaction conflict (HVIC) alarm algorithms predominantly rely on the bounding box overlap ratio. This criterion often fails in complex industrial environments, leading to excessive false positives caused by stationary vehicles, perspective distortion, and boarding/alighting activities. To address this limitation, this study proposes a dual-stage intelligent agent architecture designed to minimize false alarms. The system integrates YOLOv8 as a front-end lightweight detector for real-time candidate screening and Qwen2.5-VL, a domain-adaptive multimodal large model, as the back-end semantic verifier. A comprehensive dataset comprising one million port-specific images and videos was curated to support a novel two-phase training strategy: image pretraining for object recognition followed by video fine-tuning for temporal logic understanding. The agent dynamically interprets alarm events within their spatiotemporal context. Field trials at an operational wharf demonstrate that the proposed agent achieves an alarm precision of 95.7% and reduces false positives by over 50% across major error categories. This approach offers a highly reliable, automated solution for industrial security monitoring.