Comparison of the accuracy of selected sentiment analysis tools with psychologists’ evaluations

This article focuses on a comprehensive evaluation of three available tools—online services (ChatGPT, Google Cloud NLP, and NLP Cloud)—for determining sentiment from text. Currently, there is a strong emphasis on understanding not only the content of the text but also the manner (style and method) in which it is written, especially in interpersonal communication. We input 20 texts with varying sentiment tones into these services using an interface. Additionally, ten psychologists examined the texts to clearly determine, from a professional and human perspective, the feeling each text evokes in a reader. From the results, we observe insignificant differences in the scores among the tools investigated. The highest score was achieved by the Google Cloud tool (0.14), while the lowest scores were achieved by the NLP Cloud (0.10) and ChatGPT (0.09) tools. The highest score of variability was identified for NLP Cloud (0.97). Statistical results simultaneously imply that there is no statistically significant difference in the sentiment scores among individual psychologists for the positive sentiment level, according to Google Cloud results. The evaluations of individual psychologists collectively form one homogeneous group in terms of the sentiment score for the positive level, according to Google Cloud results. While repeated-measures tests were non-significant and confidence intervals overlapped across tools in our corpus, these findings indicate within-sample similarity rather than general interchangeability. We therefore refrain from interchangeability claims and interpret the results as no detected differences in this dataset, conditional on the texts, domains, and scoring scale used.