LLMs are not for everything: A Dataset and Comparative Study on Argument Strength Classification

A crucial task in computational argumentation is the classification of arguments into categories according to their strength.This work explores the challenge related to this task.It introduces a novel dataset of 615 curated arguments extracted from the CreateDebate platform, chosen for their diversity and relevance to a range of controversial topics.These arguments were annotated for strength into three categories, "weak," "moderate," and "strong"; Following the annotation process, a rigorous multi-step process was applied to ensure data reliability.The research compares three AI approaches to predict the strength of the argument, a) fine-tuned classification using BERT, b) fine-tuned regression using RoBERTa, and c) prompt-based classification with GPT-4.The results suggest that the regression model significantly outperforms both the fine-tuned classification model and the prompt-based LLM approaches, offering a more accurate and nuanced understanding of the strength of the argument.Despite the results mentioned above, all approaches still face challenges in capturing subtle nuances, highlighting some key areas for future research; Those can include the integration of more sophisticated discourse-aware features, and the development of enhanced representation techniques for argument strength prediction, providing a foundation for future research for automatic argument classification systems.