EFFICIENT UNIMODAL AND MULTIMODAL VIDEO UNDERSTANDING

Video plays a pivotal role in contemporary life and constitutes a major media type on the internet, with vast quantities of content uploaded every minute. Video understanding, which entails recognizing and localizing actions or events in videos, is critical for applications such as video recommendation, content analysis, autonomous driving, and assistance for the elderly or visually impaired. Despite its importance, the high-dimensionality of video data presents significant challenges, notably excessive GPU memory consumption and high computational costs. This thesis addresses these challenges by proposing algorithms aimed at enhancing memory efficiency, computation efficiency, and model efficiency for both unimodal and multimodal video understanding tasks. We introduce a memory-efficient algorithm Stochastic Backpropagation to enable end-to-end training with limited GPU memory. To tackle the high computational cost, we combine our proposed Stochastic Backpropagation with a memory bank, dubbed as TallFormer, significantly reducing training time and resource consumption for the temporal action localization task. Additionally, we demystify complex video-and-language pretraining, providing a recipe for efficient and effective video-and-language pretraining. Finally, we address the growing need for parameter-efficient tuning in large VidL models to accommodate domain shifts without catastrophic forgetting. Our proposed DaM method outperforms previous models in continual learning settings, demonstrating significant improvements in accuracy and reduced forgetting across diverse video question-answering datasets. These advancements aim to democratize video understanding research, making it more accessible to researchers with limited computing resources and accelerating progress in this critical field.