LLM-RuleVST: Rule-guided vessel trajectory and collision-risk prediction in inland waterways using UAV imagery

Inland-waterway monitoring relies largely on the Automatic Identification System (AIS), but data continuity and timeliness depend on shipborne equipment and shore-based communication infrastructure. Unmanned aerial vehicles (UAVs) can complement AIS by providing flexible, high-resolution visual observations. This study proposes the Large Language Model Empowered Rule-Guided Visual-Sequence Transformer (LLM-RuleVST), a Transformer-based framework that combines UAV visual perception with navigational rule knowledge for vessel trajectory and collision risk prediction. The framework detects and tracks vessels in UAV imagery, refines their image-plane centers, transforms the detections into world coordinates to generate UAV time-series data (UAV-TSD), and employs an LLM to retrieve and reason over a hierarchical rule base. The resulting structured rule features are fused with vessel-motion sequences through sequence-level cross-attention to predict future trajectories and continuous collision risk index (CRI) values. LLM-RuleVST was evaluated on processed AIS and UAV-TSD datasets. On the UAV-TSD dataset, its mean final displacement error (FDE) was 46.60 m, compared with 58.70 m for the strongest selected trajectory baseline, representing a reduction of 12.10 m, equivalent to 20.6%. Its mean high-risk-classification F1-score was 0.9316, compared with 0.8845 for the strongest selected baseline, representing an increase of 0.0471, equivalent to 5.3%. The source code is available at https://github.com/popqiezi/LLM-RuleVST .