From open-vocabulary detection to structural assessment: An agentic federated approach for guardrail monitoring

This work presents an agentic, federated AI framework for automated roadside guardrail monitoring across the Edge–Cloud Continuum, integrating real-time Open-Vocabulary Detection (OVD) on resource-constrained edge devices with privacy-enhancing federated adaptation of parameter-efficient Large Language Models (LLMs) for structured inspection reporting. At the edge, a promptable You Only Look Once (YOLO)-E detector identifies guardrails and potential defects with low latency, while Parameter-Efficient Fine-Tuning (PEFT) adapters enable on-device LLM fine-tuning and generation of maintenance summaries without transferring raw imagery. To improve robustness under visually degraded conditions, a Light Detection and Ranging (LiDAR)-based structural assessment module that provides geometry-centric validation and quantitative deformation indicators derived from point clouds is integrated. An agentic control plane orchestrates sensing, inference, and learning across heterogeneous edge nodes by performing energy- and Quality of Service (QoS)-aware decision-making: based on environmental context, uncertainty estimates, and resource constraints, it dynamically selects RGB, LiDAR, or fused sensing pipelines and coordinates Federated Learning (FL) rounds via secure aggregation. From a system perspective, the agentic control plane explicitly targets the reduction of energy consumption and communication costs while maintaining environmental resilience under adverse conditions, prioritizing adaptive modality selection rather than maximizing perception fidelity at all times. Experimental results show that open-vocabulary edge inference achieves real-time performance, federated PEFT substantially reduces synchronization overhead, and adaptive multimodal activation improves recall in challenging scenarios while limiting LiDAR usage, yielding a favorable robustness–energy trade-off for sustainable infrastructure monitoring.