Using LLMs for Annotation and ML Methods for Comparative Analysis of Large-Scale Turkish Sentiment Datasets

With the increasing prevalence of e-commerce plat-forms, understanding customer sentiments expressed in product reviews is crucial for assessing platform and product performance. However, traditional sentiment analysis (SA) models often struggle with low-resource languages such as Turkish, particularly due to the language's agglutinative nature and morphological complexity. To address this, several public datasets are now available online. However, these datasets are often annotated using rule-based methods, resulting in poor accuracy for Machine Learning (ML) models. This study addresses these challenges by employing two large Turkish datasets, TRSAv1 and VSCR, and improving their sentiment annotations through advanced Large Language Models (LLMs) such as ChatGPT4o-mini. The reannotation significantly enhanced the performance of traditional ML models-Logistic Regression, Naive Bayes, Random Forest, and Support Vector Machine-especially in the neutral and negative sentiment classes, as evidenced by increased precision, recall, and F1-scores. The findings underscore the importance of thorough data preparation and demonstrate that utilizing LLMs for sentiment annotation in Turkish can lead to more reliable SA, providing a stronger foundation for developing advanced SA tailored to low-resource languages. This study is the first to employ recent LLM for relabelling Turkish customer reviews, highlighting the potential for improved accuracy in SA for low-resource languages.