The wisdom of the lexicon crowds: leveraging on decades of lexicon-based sentiment analysis for improved results

The “wisdom of the crowd” (WoC) refers to the notion that collective human knowledge is capable of outperforming even individual expert knowledge. This study investigates the application of this phenomenon to lexicon-based sentiment analysis of text data. Lexicons are frequently used to classify the sentiment of text data, particularly in the absence of sentiment class label information. We propose leveraging some of the most popular, publicly-available lexicons created in the last half century to improve sentiment analysis performance. Specifically, this research argues that the collective information provided by the thirteen lexicons included in the crowd constitutes a WoC situation that can more accurately predict the sentiment in the majority of example cases when compared to individual lexicons, lexicon ensembles, and machine learning methods. Thirteen popular sentiment-labeled text datasets, comprised of different types of text data and covering a variety of domains, are used to test this research proposition. We show that the WoC sentiment analysis achieves greater performance than individual lexicons, which are considered to be ‘experts’, and a lexicon ensemble approach. In comparing our novel approach to sentiment analysis against popular machine learning approaches, the proposed WoC method achieves superior results in the majority of examples. By overcoming many of the limitations of other approaches with high accuracy, the WoC method can provide organizations with real-time, reliable, and accurate sentiment analysis.