Chinese Vision-language Understanding based on Multimodal Pre-trained Models

The purpose of Chinese Vision-Language Understanding (CVLU) task is to enhance the capabilities of Chinese vision-language multimodal pre-trained models in vision-language multimodal modeling and understanding. The complex semantics and ambiguity of Chinese language result in varying performance levels of the general pre-trained models across different tasks. We verify the different performances of the general pre-trained model in processing vision-language information from different perspectives through four subtasks, including image-text retrieval, visual question answering, visual grounding and visual dialog. At the same time, to solve the above problems, we use task-specific datasets to fine-tune the general pre-trained model, enhancing its adaptability to various subtasks. The experimental results show that most of the evaluation metrics for the four subtasks have improved by over 10%, validating the efficacy of the proposed approach.