Assessing the Performance of Pretrained Models for Accurate and Consistent Classification of Argument Structure Constructions

Despite the growing body of (applied) linguistic research on argument structure constructions (e.g. ditransitive: He gave me a ball), few studies have attempted to develop a classification model for these constructions. Furthermore, existing models struggle with inconsistent performance across different constructions and datasets, highlighting the need for improvement. The current study explores whether broadening and improving the quality of training data can enhance the performance of deep learning models in classifying constructions. Specifically, the study fine-tunes two pre-trained models, BERT and RoBERTa, and examines the errors they produce. The results show that using a high-quality dataset that includes diverse genres/modalities improves performance accuracy and consistency, and RoBERTa (F1 = 0.925) outperforms BERT (F1 = 0.916) in this context. The error analysis further reveals that the two models exhibit similar error patterns, but with distinct strengths: The BERT-based model outperforms in intransitive resultatives, while the RoBERTa-based model outperforms in caused motions and intransitive motions. Overall, these findings highlight the importance of using diverse, high-quality datasets and selecting the optimal model architecture to maximize performance, while also revealing areas for future improvement. Additionally, the high accuracy of our systems suggests their potential to make a meaningful impact in educational and assessment settings.