Method of Video Key Frame Extraction based on Lightweight-SAM

Keyframe extraction is a very important part of the video retrieval steps. However, current keyframe extraction techniques have problems such as underrepresentation, large redundancy, poor adaptability and large computation. So, we propose an accelerated alternative to the basic task with comparable performance, and by rephrasing the task as fragment generation and prompting. We use the key frame extraction method combined with the SAM-based feature extraction model of lightweight-SAM, without additional training, to obtain an interactive video key frame extraction method for focused preference objects. In this model, we used the Yolov8-seg model to replace the VIT part in Segment-Anything Model (SAM), while adding the tracking module XMem in the keyframe extraction step, which can effectively reduce the computational amount while also ensuring the accuracy of the keyframes. Using our approach, we achieve comparable performance to the SAM approach with a 50x higher runtime speed. Recall at 50% precision (R50) is high and has significantly fewer parameters (only 70M). The ratio of key frame extraction can reach 90.3% under the premise of an average fidelity of 69.4%. We compare the performance of these methods on UCF-101, COCO, BSDS500 and other datasets, and this interactive design performs impressively in video object tracking and segmentation, and performs well in keyframe extraction for selected objects.