Expert comment generation from sports videos using multimodal LLM

This paper presents a method for generating expert comments on human motion in sports videos using a Multimodal Large Language Model (MLLM). In the proposed method, a pretrained Vision Transformer (ViT) encoder and a transformer encoder realize the extraction of tokens from sports videos, enabling expert comment generation by considering temporal information. Experiments using basketball videos in the Ego-Exo4D dataset validated the effectiveness of incorporating temporal information in expert comment generation, demonstrating the proposed method’s superiority over existing techniques.