Multi-modal smart classroom topic segmentation

In recent years, with the development and progress of technology, new models of smart classrooms have been introduced into classrooms. With the popularization of new models, rich teaching resources have been generated through the recording of smart classrooms. There are two problems in topic segmentation in smart classroom environment. The one is the transcription errors of automatic speech recognition (ASR) system, which affects the topic segmentation. On the other hand, teachers need to constantly maintain classroom discipline, which makes the classroom content appear incoherent, which increases the difficulty of topic segmentation. To address the above problems, this paper proposes a multi-modal smart classroom topic segmentation model. The model obtains the corresponding text information from the video frame through optical character recognition (OCR) technology, and obtains the speech transcription text from the classroom audio through the automatic speech recognition (ASR) system. They are sent to the text encoder for encoding, supplemented by the cross-modal attention mechanism, and finally outputs the probability of each transcribed text as the smart classroom chapter segmentation point through BiLSTM and MLP. The model adds dynamic OCR recognition and similarity detection deduplication algorithm to improve the processing speed for video frame information. Experiments show that the accuracy and scalability of the solution proposed in this paper are significantly better than the baseline method in the domain. The dynamic OCR recognition and similarity detection deduplication algorithm greatly reduces the processing time of the model.