Divide-and-Summarize: Enhancing Deep Neural Video Summarization

Sequence-based neural architectures, such as Long Short-Term Memory (LSTM) networks and Transformers, have driven advances in supervised video summarization by modeling inter-frame dependencies. However, existing methods assume that long-range dependencies are essential for summary generation, which may lead to unnecessary computational overhead. To address this, we propose a Field of View (FOV) adjustment strategy, Divide-and-Summarize (DIV-SUM). By partitioning input videos into smaller fragments of predefined size, our approach explicitly models short-range inter-frame relationships, enabling a fully parallelizable end-to-end video summarization pipeline. Furthermore, most prior work formulates neural video summarization as a frame-wise score regression task. We introduce a simple yet effective target space quantization module, which discretizes the regression targets into classes, introducing a tolerance margin that improves performance. Our approach offers two key benefits: (1) we achieve state-of-the-art performance on the SumMe benchmark while remaining competitive on TVSum, and (2) we significantly reduce the computational cost of inference, improving efficiency without sacrificing quality.