Deep Reciprocal Learning for Image Captioning

The current training strategies based on knowledge distillation for image captioning assume that each learning model possesses complete learning value, lacking review and guidance mechanisms among the interactive process of models. To address this problem, we propose a novel Captioner with Deep Reciprocal Learning (CaDReL) for image captioning inspired by the social learning theory, which realizes interactive learning between models controlled by salient semantic evaluation. In CaDReL, we analyze the semantic saliency of each learning network to better control the parameter transfer in knowledge distillation by cyclically alternately freezing and unfreezing two learning networks with identical review mechanisms. We also propose a novel cascade bridging diffusion module, which fuses feature information from different levels of visual information and attention ranges in the encoder by a cascade diffusion mechanism to capture rich image details and contextual information. Meanwhile, an attention guided knowledge augmentation module is proposed to guide knowledge transferring by the attention maps from the respective encoders of two peer joint learning modules for improving the robustness of the whole training strategy. Experimental results illustrated that the proposed CaDReL achieves excellent performance on the MSCOCO dataset, and outperforms most state-of-the-art methods. Codes are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/ZJ-VIP-Lab/Deep-Reciprocal-Learning-for-Image-Captioning</uri>.