Real-time Multi-modal Highlight Prediction for Simultaneous Viewing of Multiple Live Streams

With the development of real-time video streaming technology on the internet, live streaming services have been popular. In live streaming, it is common for users to have multiple live streams they want to watch simultaneously. Live streaming allows for simultaneous viewing across multiple windows. However, when engaging in simultaneous multi-window viewing, there is a risk of missing highlights from one stream while focusing on another. In this study, to avoid missing highlights during simultaneous viewing, we propose a method that predicts upcoming highlights by utilizing weighted moving averages that can capture trend changes in time series data in real time, based on chat text and audio from live streams. We evaluated the effectiveness of the proposed method using 23 YouTube Live streams, confirming that combining both textual and audio information enables effective real-time highlight prediction.