BRWF-Captioner: A Novel Better-Refine method for Image Captioning with Word Frequency Noise Schedule

Diffusion model has achieved outstanding generative capability in image captioning task. However, in the text noise addition process, most of the methods use linear or cosine noise schedule, resulting in the same level of noise being added to each token in the text. Meanwhile, it would be unreasonable to use the same inference step for Easy-to-Recover Tokens (ERT) and Hard-to-Recover Tokens (HRT) in the inference process, with previously predicted incorrect tokens that will not be changed. To address the issues, this paper proposes a Novel Better-refine method for Image Captioning with Word Frequency Noise Schedule. Firstly, a Word Frequency Noise Schedule (WFNS) is proposed, according to each token the amount of information it carries to control how much noise is added to it to enhance the consistency of the model for text tokens. In addition, in order to improve the quality of sentence generation, this paper also design a Better-refine (BR) method, i.e., using ERT to serve as a condition to guide the remaining tokens to continue recovery and refinement. The whole model is abbreviated as BRWF-Captioner. Finally, this paper validates the effectiveness of the method on the MS COCO dataset. BRWF-Captioner improves the Bleu-4 and CIDEr metrics by 1.3% and 1.8%, respectively, and demonstrates competitive performance with state-of-the-art models (e.g. DDCap, Bit Diffusion).