EdgeVidCap: A Channel-Spatial Dual-Branch Lightweight Video Captioning Model for IoT Edge Cameras

With the deep integration of edge computing and Internet of Things (IoT) technologies, the computational capabilities of intelligent edge cameras continue to advance, providing new opportunities for the local deployment of video understanding algorithms. However, existing video captioning models suffer from high computational complexity and large parameter counts, making them challenging to meet the real-time processing requirements of resource-constrained IoT edge devices. In this work, we propose EdgeVidCap, a lightweight video captioning model specifically designed for IoT edge cameras. Specifically, we design a hybrid module termed Synergetic Attention State Mamba (SASM) that incorporates channel attention mechanisms to enhance feature selection capabilities and leverages State Space Models (SSMs) to efficiently capture long-range spatial dependencies, achieving efficient spatiotemporal modeling of multimodal video features. In the caption generation stage, we propose an adaptive attention-guided LSTM decoder that can dynamically adjust feature weights according to video content and auto-regressively generate semantically rich and accurate textual descriptions. Comprehensive evaluations of EdgeVidCap on mainstream datasets, including MSR-VTT and MSVD are analyzed. Experimental results demonstrate that our system demonstrated enhanced precision relative to existing investigations, and our streamlined frame filtering mechanism yielded greater processing efficiency while creating more dependable descriptions following frame selection.