Commentary Master: Exploring Fine-grained Video Action Commentary

Commentators can provide rich information and analysis to bring good experience to the audience, leading to great application prospects for AI commentators. In this paper, we propose Fine-grained Video Action commentary (FVAC) as a new challenging task that uses long natural language to comment the action in the input video like a real commentator. In addition to describing the action, FVAC focuses on providing detailed evaluation, background knowledge, and technical instructions. Due to the high difficulty to obtain detailed semantic information in the video and the lack of background knowledge, existing researches are based only on action evaluation and description. Fortunately, the great achievements of AI agent in general artificial intelligence give us new solutions. To solve the task, we propose a strong AI agent called the Commentary Master framework to generate fine-grained commentary. In the agent, we propose Discriminator Net (DNet) to extract detailed semantic information and Balance Tree Retrieval-Augmented Generation (BTRAG) to provide rich background knowledge and examples. Additionally, to address the difficulty of evaluating the commentary content with rich styles, we propose the Commentary Quality Level (CQL) metric. Compared to base LVLM models, our experiment shows that the Commentary Master can generate high-quality fine-grained commentary.