Video understanding with image, audio, and text

With the explosive growth of streaming video data, describing or understanding these videos has become an interesting topic within the international academic community. However, existing methods have ignored the important information among the image, audio, and text, resulting in insufficient understanding of the video. In this paper, we propose a novel video understanding algorithm that incorporates the above neglected information. Firstly, this method combines speech recognition and a Large Language Model(LLM) to obtain the detailed textual descriptions of the video. Secondly, the image and textual descriptions are combined to obtain video keyframes. Finally, the textual descriptions and keyframes are concatenated to gain pivotal video understanding results. Extensive experiments have shown the superiority of the proposed method.