Camscribe: Enhanced Dashcam Video Descriptions Through Multimodal Spatiotemporal and Object Detection for Autonomous Vehicles

The generation of accurate and coherent video descriptions necessitates comprehensive understanding of multiple visual cues. While conventional video description models have predominantly relied on RGB and optical flow information, yet these approaches face fundamental accuracy constraints, underscoring the importance of exploring expanded modalities in both visual and audio domains. This study introduces a novel approach that integrates three distinct visual modalities, i.e., spatial features, temporal dynamics, and object-level semantics, to enhance description quality through the incorporation of fine-grained object information, resulting in more effective and semantically meaningful outputs. Our framework proposes the integrated YOLO object detection for multilabel class embeddings generation, combined systematically with spatial and temporal embeddings through an early fusion approach. These integrated embeddings are then processed using memory-augmented extended recurrent transformer architecture capable of handling upto 14 sentences recursively with 12 attention heads for complex sequence modeling. We also propose a Harmonic Aggregation of Language Evaluation Metrics (HA-LEM) for model selection that combines multiple linguistic dimensions, enabling robust model selection during the training phase and resulting in enhanced paragraph generation capabilities. The model training was conducted on the DeepRide dashcam dataset, leveraging multimodal video features for robust performance. The efficacy of our proposed methodology is validated through extensive qualitative, quantitative evaluation and ablation analysis. Our framework achieves superior performance on the DeepRide dataset, as evidenced by comprehensive evaluation using standard metrics: BLEU@4 (35.02), METEOR (27.92), ROUGE-L (45.78), and CIDEr (68.82), surpassing previous state-of-the-art results.