Multimodal deep learning framework for soccer video event detection and summarization

This paper presents a novel multimodal deep learning framework for soccer video analysis and summarization. The framework comprises four main modules: the Video Event Detection Module, the Keyword Spotting Module, the Crowd Cheering Classification Module, and the Video Summarization Module. The Video Event Detection Module integrates two submodules: the Shot Boundary Detection Submodule, which segments the match video into temporal shots, and the Video Classification Submodule, which classifies detected shots into soccer events. Trained on the SoccerVE dataset, this module achieves $$90.8\\%$$ accuracy in soccer event detection, outperforming the previous state of the art by 6.2 percentage points. The Keyword Spotting Module, trained on the SoccerAE dataset, detects match-related keywords from commentator audio and obtains $$85.8\\%$$ accuracy with fewer parameters than comparable models. The Crowd Cheering Classification Module, trained on the SoccerCC dataset, classifies spectator reactions and reaches $$98.7\\%$$ accuracy, effectively capturing match atmosphere. The Video Summarization Module integrates the outputs of the three event detection modules (Video Event Detection, Keyword Spotting, and Crowd Cheering Classification) and applies a scoring function with event weighting and knapsack optimization to generate concise and context-aware match summaries. Evaluation across ten complete matches demonstrates an overall summarization precision of $$91.0\\%$$ , representing a significant improvement over previous state-of-the-art methods. These results highlight the potential of multimodal deep learning to deliver high-precision and context-aware insights for sports analytics and automated highlight generation.