Enhancing sentiment analysis accuracy: strategic pre-calibration to mitigate anchoring bias

Abstract In this study, we explore the concept of anchoring bias in the context of sequential sentiment analysis of review corpora. We introduce a novel approach that involves using a carefully selected, limited group of reviews at the beginning of the annotation process for calibration, aiming to reduce this bias. Through an extensive set of experiments we confirm the existence of sentiment bias and demonstrate that indeed its impact can be moderated through initial calibration. We also demonstrate that the composition of the calibration set is critical, underscoring the importance of establishing sound criteria for selecting these initial reviews. By comparing the accuracy of annotators who utilized our calibration method against those who did not calibrate or used a randomly chosen set for calibration, we found that our method significantly decreases the overall annotation error. Moreover, the guidelines we develop for selecting the calibration set prove to be highly effective and adaptable, even when applied to a domain other than the one they were originally developed for. Acknowledging the overhead of labeling calibration reviews, we demonstrate that this approach is more efficient compared to eliciting multiple sentiment scores per review, a common strategy to reduce Mean Absolute Error (MAE). Our findings reveal that the proposed calibration method significantly reduces resource expenditure, compared to relying on parallel labeling, while maintaining accuracy.