Combination of Fine-Tuned GECToR and BERT Language Models for Improving Spelling Error Correction

Grammar Error Correction (GEC) is a challenging task, often focusing on correcting all or one specific type(s) of errors. Among these, spelling error is widespread across different levels of language learners. In this paper, we aim to enhance spelling error correction for GECToR, a typical GEC model, by proposing a set of methods. Firstly, a fine-tuned GECToR named GECToR FS-FT is introduced following two sub-steps. In the first sub-step, a spelling dataset is added to a multiple-error dataset named W&I+LOCNESS to create W&I+LOCNESS-SpellAdd. The spelling dataset is taken from 1BW, an augmented spelling dataset benchmark, by conducting a repeating fine-tuning to identify its optimal amount. Then, GECToR's training process is improved by adding an initial training stage on PIE-synthetic dataset, and in stage 2, the model is fine-tuned on W&I+LOCNESS-SpellAdd dataset. Secondly, a model specifically designed for spelling correction named BERT is utilized. The improved spelling correction of BERT is based on an observation of frequent error patterns on about 50% of the data amount of W&I+LOCNESS and then a set of rules are introduced to correct the mistakes. Lastly, the outputs of improved BERT's corrections are combined with those of GECToR FS-FT following the rule of higher priority for GECToR FS-FT than BERT. Evaluation on BEA-2019 (dev) dataset shows a significant improvement in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">${F_{0.5}}$</tex> score for spelling error correction from 4.13 to 6.82 higher than those of some previous GECToR-based methods.