Temporal-invariant video contrastive learning: A novel perspective from brain knowledge

Contrastive learning, as a powerful self-supervised learning method, has made significant strides for images. However, previous studies for videos concentrate on practical performance improvements and yield remarkable results, while still lacking more in-depth theoretical exploration behind these advancements. Especially in light of the additional temporal dimension in videos compared to images, the importance of developing a novel perspective to rethink introducing new knowledge for dynamic-video is highlighted, which further supports the improvement of the learned representations. Therefore, this paper aims to explore a novel intellectual perspective, which injects new insights into video contrastive learning to further promote technical progress. For that, we delve into brain knowledge to redefine the essential temporal information (called temporal-invariant information) and temporal-redundancy in videos, subsequently exploring the construction of temporal views inspired by the attentional modulation mechanism as Temporal-invariant Video Contrastive Learning (TiVCL). Specifically, four temporal-invariant strategies are proposed to disturb temporal-redundancy, corresponding to distinct cognitive characteristics in human top-down attention mechanism (i.e., attention competition, perceptual constancy, motion association, and contour integration), providing preliminary validation for the heuristic value of single cognitive characteristics. Furthermore, we synergistically integrate these four strategies into an integration scheme, demonstrating the crucial role of multi-cognitive characteristics collaboration in extracting temporal-invariant information in video contrastive learning. Empirical experiments on multiple video benchmarks are performed to validate the optimal parameters of the temporal views generated through both individual strategies and the integration scheme in the proposed TiVCL. Moreover, excellent results across different video tasks also demonstrate the superior performance of TiVCL over advanced video methods. This study attempts to draw a more expansive picture for motivating more promising brain-inspired works for videos.