Automated Multimedia Objects Extraction and Text Generation from Lecture Videos Using Deep Learning Models

Today, online education platforms are the main source of knowledge, especially lecture videos. Watching a long lecture video to grasp its content is time-consuming and inefficient. As online video content continues to grow, the need to break down complex multimedia content is increasing. In this paper, a different approach is presented to extract visual and audio information from a lecture video, using speech recognition to generate accurate transcriptions and frame segmentation to capture important visual elements. Visual elements include texts, diagrams, and images, which contain valuable information. From these visual objects, information is extracted in textual form and then rephrased and structured to generate a textual document. A new data set of lecture slide images was introduced for frame object detection to enhance this process. This was achieved through various steps such as video frame selection, frame object detection, frame text extraction, audio transcript extraction, and text generation. Comparisons were made between rule-based approaches and deep learning approaches, and the results showed that deep learningbased approaches outperformed rule-based techniques in accuracy and efficiency. These findings show that deep learning-based multimodel approaches enhance learning for students with visual or hearing impairments, improve search and navigation, and enable text-based compression of lecture videos.