Towards the design of hybrid transformer-driven deep representation learning for high-performance video retrieval systems

Abstract With the advancement of digital media technologies, the rapidly increasing amount and intricacy of multimedia data cause numerous issues in querying, processing, and storing data on time. The processing time and feature extraction play a very significant part in large-scale video retrieval systems and are presently receiving much consideration from scholars. With the rising volume of video data accessible, conventional video retrieval models are frequently inadequate to tackle the requirement for more accurate and precise retrieval. In recent times, convolutional neural networks (CNNs) have shown superior ability in video retrieval, because they can efficiently extract and learn higher-level representations from video frames. At present, numerous models are developed for the extraction of features from files and recovering videos according to their content features. The video content feature could be extracted, containing objects, motion, speech, and much more. The scientific task set for these analyses is to resolve the best balance between fast response time and high accuracy, particularly in larger data processing models. Furthermore, a massive volume of video data is frequently needed to train and implement beneficial machine learning (ML) models in entertainment and industry. Small organisations don’t have the amenity of retrieving sufficient data for ML due to the limitation of computational resources. Therefore, this study develops a hybrid transformer deep learning-based High-Performance Video Retrieval (HTDL-HPVR) model. The proposed model aims to retrieve videos effectively via the integration of shot boundary detection, key frame extraction, and a deep ranking model to improve retrieval precision and speed. In the proposed model, shot segmentation is performed by using a Hybrid Transformer-Convolutional Network (HTCN) model to capture spatial and temporal transitions proficiently. Next, the Vision Transformer (ViT)-based key frame extractor recognises representative frames via optimal visual diversity and removes redundancy. In addition, the Siamese ranking network (SRN) method further refines similarity scores between query and candidate videos, enhancing retrieval accuracy. Finally, the similarity score outcomes are used in the classification process through a convolutional autoencoder (CAE) method to classify the video frames. The experimental evaluation of the proposed method is tested using the benchmark UCF101 dataset from the Kaggle repository. The dataset comprises numerous camera motions, scales, lighting conditions, and viewpoints. Extensive comparative studies highlighted the betterment of the presented model over other existing models in terms of diverse measures. Therefore, the HTDL-HPVR approach is applied as a robust tool for a high-performance video retrieval process.