Medium-Grained Semantics Uncertainty Correction for Text-Video Retrieval

Text-Video retrieval is a challenging research task in the multimedia field. Current methods either do not consider one-to-many uncertainty or do not fully consider the inherent multi-level semantics of videos and texts. In this paper, we propose a new human-like method to construct medium-grained semantics from coarse-grained and fine-grained semantic features in a top-down and bottom-up manner, and use medium-grained semantics to correct uncertainty in baseline models. For bottom-up medium-grained semantic construction, we propose an adaptive semantic aggregation (ASA) module. We introduce learnable labels in the encoder to adaptively aggregate these medium-grained semantics, and then represent fine-grained semantics and medium-grained semantics as probability distributions to model uncertainty. For top-down medium-grained semantic construction, we propose a global semantic disentanglement (GSD) module that disentangles coarse-grained global semantics into medium-grained semantics and performs uncertainty matching with medium-grained semantics. We believe that these two modules mimic the process of human reasoning from local to global and from global to local. We conducted extensive experiments on MSR-VTT, LSMDC and MSVD datasets, and the results showed that our method outperforms the existing state-of-the-art methods.