MKER: multi-modal knowledge extraction and reasoning for future event prediction

Humans can predict what will happen shortly, which is essential for survival, but machines cannot. To equip machines with the ability, we introduce the innovative multi-modal knowledge extraction and reasoning (MKER) framework. This framework combines external commonsense knowledge, internal visual relation knowledge, and basic information to make inference. This framework is built on an encoder-decoder structure with three essential components: a visual language reasoning module, an adaptive cross-modality feature fusion module, and a future event description generation module. The visual language reasoning module extracts the object relationships among the most informative objects and the dynamic evolution of the relationship, which comes from the sequence scene graphs and commonsense graphs. The long short-term memory model is employed to explore changes in the object relationships at different times to form a dynamic object relationship. Furthermore, the adaptive cross-modality feature fusion module aligns video and language information by using object relationship knowledge as guidance to learn vision-language representation. Finally, the future event description generation module decodes the fused information and generates the language description of the next event. Experimental results demonstrate that MKER outperforms existing methods. Ablation studies further illustrate the effectiveness of the designed module. This work advances the field by providing a way to predict future events, enhance machine understanding, and interact with dynamic environments.