A Study on Automatic English Grammatical Error Correction Using Transformer and BERT

This study evaluates grammatical error correction using Transformer and pre-trained BERT models. The goal is to understand how regulating different parameters can affect the effectiveness of these two models in mitigating errors in an English text. This study involves a comparative analysis of these two models across different experimental aspects. The impact of various factors on model performance by using different dataset sizes, dataset types, and fine-tuning parameters such as learning rates and the number of epochs has been explored. Our findings show that the base Transformer model generally outperformed the BERT model in every experimental aspect. The base Transformer model appears to slightly outperform BERT in terms of achieving the highest GLUE score of 79.31 percent compared to BERT’s 78.76 percent. However, the BERT model performs well with the user-annotated dataset compared to the synthetic one. This experimental study concludes that the base Transformer model is highly effective at capturing long-range dependencies and parallelizing training, which makes the model more effective than the BERT model in the context of Grammatical Error Correction. Future work could address the large dataset size and evaluate different corpora to enhance result analysis.