LLM-based Evaluation of LLM Generated, Detail Oriented Video Material Summaries Services

This paper addresses the gap in summarization quality assessment, particularly in the context of instructional video materials, which have been largely overlooked in existing benchmarks dominated by written text. We introduce eight specific criteria tailored to evaluating the quality of instructional video summaries, recognizing the distinct characteristics of such content and the importance of facilitating users' access to relevant instructional materials. Leveraging large language models for speech-to-text processing, summary generation, and quality assessment, we conducted a comprehensive study. We outline our experimental methodology and present our findings, which confirm the effectiveness of language models in generating optimal textual summaries. Our proposed criteria, including "Narration," "Sources," and "Examples," demonstrate significant enhancement when applied to summaries generated with additional instructions. However, general criteria did not exhibit improvement in summary quality when using advanced prompts compared to baseline methods. Lastly, we discuss the limitations of our research approach and highlight areas for future investigation.