RMVSpell: Chinese Spelling Check Based on Multimodality and Image Input Enhancement

Chinese Spelling check (CSC) is a traditional NLP task that aims to detect and correct spelling errors in Chinese texts and has a wide range of applications. Most Chinese spelling errors are caused by the misuse of pinyin and glyphs. Previous researchers have also conducted a lot of research on how to introduce pinyin and glyph information into the CSC task, but they did not use real image data for research. Therefore, we constructed a real image dataset and introduced it, and adopted a multimodal approach for training. Then we use Bert to extract semantic features, GRU to generate phonetic information and Transformer to extract phonetic features, and Visiontransformer to extract visual features. In order to enable the model to better fuse features from different modalities, we also propose a multi-level post-fusion network called RMVSpell to fuse features from different modalities at multiple levels. The fused features are then passed into the output classification layer to obtain the final error correction result. The experimental results on the SIGHAN15 test set show that the performance of our model is better than most baseline models, and the F1 score achieved the best results, which illustrates the effectiveness of our method.